[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:45:36 dist-test-slave-jpj1 krb5kdc[28498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:45:36 dist-test-slave-jpj1 krb5kdc[28498](info): set up 2 sockets
Aug 09 06:45:36 dist-test-slave-jpj1 krb5kdc[28498](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 06:45:37.191998 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:45:37 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:45:37.368760 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44315
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:37.690531 28520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:37.690817 28520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:37.690935 28520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:37.697925 28520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:45:37.698035 28520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:37.698108 28520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:37.698182 28520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:45:37.698241 28520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:45:37.707821 28520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:37.709076 28520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:37.710354 28520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:37.721845 28529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:37.721903 28528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:37.721942 28533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:37.722245 28520 server_base.cc:1034] running on GCE node
I20260809 06:45:37.723176 28520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:37.724567 28520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:37.725800 28520 hybrid_clock.cc:648] HybridClock initialized: now 1786257937725744 us; error 40 us; skew 500 ppm
I20260809 06:45:37.726135 28520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:37 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257937, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:37.729211 28520 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:45:37.730626 28520 webserver.cc:460] Webserver started at http://127.27.209.254:35145/ using document root <none> and password file <none>
I20260809 06:45:37.731293 28520 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:37.731412 28520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:37.731745 28520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:37.734805 28520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "d6083a9881ec44eb9ff829465d44e72f"
format_stamp: "Formatted at 2026-08-09 06:45:37 on dist-test-slave-jpj1"
server_key: "57f1326270cdbe3cfeae4ab5cec18f9b"
server_key_iv: "804b231ab102ca72650fb5e1be009aa9"
server_key_version: "encryptionkey@0"
I20260809 06:45:37.735584 28520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "d6083a9881ec44eb9ff829465d44e72f"
format_stamp: "Formatted at 2026-08-09 06:45:37 on dist-test-slave-jpj1"
server_key: "57f1326270cdbe3cfeae4ab5cec18f9b"
server_key_iv: "804b231ab102ca72650fb5e1be009aa9"
server_key_version: "encryptionkey@0"
I20260809 06:45:37.740665 28520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:45:37.744165 28542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:37.745456 28520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:45:37.745616 28520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "d6083a9881ec44eb9ff829465d44e72f"
format_stamp: "Formatted at 2026-08-09 06:45:37 on dist-test-slave-jpj1"
server_key: "57f1326270cdbe3cfeae4ab5cec18f9b"
server_key_iv: "804b231ab102ca72650fb5e1be009aa9"
server_key_version: "encryptionkey@0"
I20260809 06:45:37.745818 28520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:37.765904 28520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:37.769794 28520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:37.770056 28520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:37.787750 28520 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:44315
I20260809 06:45:37.787741 28629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:44315 every 8 connection(s)
I20260809 06:45:37.789095 28520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:45:37.792761 28630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:37.795395 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28520
I20260809 06:45:37.795742 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:45:37.798213 28487 external_mini_cluster.cc:1428] Setting key 7ddb18485ae79416d484609fe4eba5b1
I20260809 06:45:37.806035 28630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f: Bootstrap starting.
I20260809 06:45:37.808974 28630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:37.809981 28630 log.cc:824] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:37.812803 28630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f: No bootstrap required, opened a new log
I20260809 06:45:37.820216 28630 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6083a9881ec44eb9ff829465d44e72f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44315 } }
I20260809 06:45:37.820585 28630 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:37.820744 28630 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6083a9881ec44eb9ff829465d44e72f, State: Initialized, Role: FOLLOWER
I20260809 06:45:37.821470 28630 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6083a9881ec44eb9ff829465d44e72f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44315 } }
I20260809 06:45:37.821761 28630 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:45:37.821918 28630 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:45:37.822125 28630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:37.824481 28630 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6083a9881ec44eb9ff829465d44e72f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44315 } }
I20260809 06:45:37.824968 28630 leader_election.cc:302] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6083a9881ec44eb9ff829465d44e72f; no voters:
I20260809 06:45:37.825556 28630 leader_election.cc:288] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:45:37.825728 28636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 1 FOLLOWER]: Leader election won for term 1
Aug 09 06:45:37 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:37.826692 28636 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [term 1 LEADER]: Becoming Leader. State: Replica: d6083a9881ec44eb9ff829465d44e72f, State: Running, Role: LEADER
I20260809 06:45:37.827345 28636 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6083a9881ec44eb9ff829465d44e72f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44315 } }
I20260809 06:45:37.827937 28630 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:45:37.830195 28637 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6083a9881ec44eb9ff829465d44e72f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6083a9881ec44eb9ff829465d44e72f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44315 } } }
I20260809 06:45:37.830565 28637 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:37.835449 28638 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6083a9881ec44eb9ff829465d44e72f. Latest consensus state: current_term: 1 leader_uuid: "d6083a9881ec44eb9ff829465d44e72f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6083a9881ec44eb9ff829465d44e72f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44315 } } }
I20260809 06:45:37.836373 28638 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:37.840824 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:37.803084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.27.209.254:44315)
0809 06:45:37.805171 (+ 2087us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:37.805185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:37.805238 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:45:37.813710 (+ 8472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:37.813733 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:37.814088 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:37.814485 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:37.817420 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:37.818903 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:37.820645 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:37.821685 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:37.821888 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:37.828364 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:37.828377 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:37.828396 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:37.828448 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:37.832806 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:37.833935 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:37.833944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:37.833953 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:37.834174 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:37.834796 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:37.834802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:37.834805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:37.835588 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:37.838682 (+ 3094us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:37.838967 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 06:45:37.839400 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":1612,"threads_started":1}
I20260809 06:45:37.842356 28641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:45:37.847872 28641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:45:37.857595 28641 catalog_manager.cc:1347] Generated new cluster ID: 191ee2321f6142ec9acbdfa6e17c7de2
I20260809 06:45:37.857705 28641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:45:37.867869 28641 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:45:37.869715 28641 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:45:37.888721 28641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6083a9881ec44eb9ff829465d44e72f: Generated new TSK 0
I20260809 06:45:37.889528 28641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:45:37.937382 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44315
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:38.270704 28669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:38.271001 28669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:38.271099 28669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:38.278355 28669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:38.278461 28669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:38.278585 28669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:45:38.288070 28669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:38.289301 28669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:38.290822 28669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:38.303611 28678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:38.303653 28679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:38.303692 28681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:38.304309 28669 server_base.cc:1034] running on GCE node
I20260809 06:45:38.304898 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:38.305792 28669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:38.307024 28669 hybrid_clock.cc:648] HybridClock initialized: now 1786257938306961 us; error 46 us; skew 500 ppm
I20260809 06:45:38.307351 28669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:38 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257938, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:38.310482 28669 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:45:38.311914 28669 webserver.cc:460] Webserver started at http://127.27.209.193:42885/ using document root <none> and password file <none>
I20260809 06:45:38.312575 28669 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:38.312695 28669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:38.312995 28669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:38.315809 28669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "60b46e233882455992147a42aa2f1973"
format_stamp: "Formatted at 2026-08-09 06:45:38 on dist-test-slave-jpj1"
server_key: "16fefbfb10728b0a8f7570b30e2f0d19"
server_key_iv: "3afa4fcb268703c71e2cadcf26357d74"
server_key_version: "encryptionkey@0"
I20260809 06:45:38.316555 28669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "60b46e233882455992147a42aa2f1973"
format_stamp: "Formatted at 2026-08-09 06:45:38 on dist-test-slave-jpj1"
server_key: "16fefbfb10728b0a8f7570b30e2f0d19"
server_key_iv: "3afa4fcb268703c71e2cadcf26357d74"
server_key_version: "encryptionkey@0"
I20260809 06:45:38.321571 28669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:45:38.324887 28689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:38.326121 28669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:45:38.326275 28669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "60b46e233882455992147a42aa2f1973"
format_stamp: "Formatted at 2026-08-09 06:45:38 on dist-test-slave-jpj1"
server_key: "16fefbfb10728b0a8f7570b30e2f0d19"
server_key_iv: "3afa4fcb268703c71e2cadcf26357d74"
server_key_version: "encryptionkey@0"
I20260809 06:45:38.326442 28669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:38.341794 28669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:38.345940 28669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:38.346204 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:38.347132 28669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:38.348516 28669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:38.348590 28669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:38.348696 28669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:38.348758 28669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:38.380141 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:34531
I20260809 06:45:38.380196 28864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:34531 every 8 connection(s)
I20260809 06:45:38.381414 28669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:45:38.385588 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28669
I20260809 06:45:38.385833 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:45:38.386607 28487 external_mini_cluster.cc:1428] Setting key 3cd4d1d13a58a120a55f5a9924052733
Aug 09 06:45:38 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257938, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:38.404388 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:38.384067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:59643 (local address 127.27.209.254:44315)
0809 06:45:38.384514 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:38.384519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:38.385602 (+ 1083us) server_negotiation.cc:407] Connection header received
0809 06:45:38.388043 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:38.388049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:38.388136 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:38.388237 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:38.390327 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.391091 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.393257 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.393659 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.393760 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:38.397664 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:38.397671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:38.397680 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:38.397723 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:38.400971 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:38.401764 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:38.401771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:38.401774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:38.401928 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:38.402442 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:38.402447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:38.402450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:38.402826 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:38.402980 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:38.403753 (+ 773us) server_negotiation.cc:299] Negotiation successful
0809 06:45:38.403990 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260809 06:45:38.405133 28870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:38.384623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44315 (local address 127.27.209.193:59643)
0809 06:45:38.385404 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:38.385466 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:38.387777 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:38.388534 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:38.388546 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:38.389286 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:38.390132 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:38.390160 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.391258 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.391263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:38.393118 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:38.393133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.393876 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.393882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:38.394157 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:38.394996 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:38.395019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:38.397397 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:38.401145 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:38.401153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:38.401166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:38.401621 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:38.402037 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:38.402040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:38.402042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:38.402321 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:38.402966 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:38.402972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:38.403325 (+ 353us) client_negotiation.cc:769] Sending connection context
0809 06:45:38.403590 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 06:45:38.404012 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
I20260809 06:45:38.406774 28865 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44315
I20260809 06:45:38.407091 28865 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:38.407938 28865 heartbeater.cc:511] Master 127.27.209.254:44315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
I20260809 06:45:38.410315 28572 ts_manager.cc:194] Registered new tserver with Master: 60b46e233882455992147a42aa2f1973 (127.27.209.193:34531)
I20260809 06:45:38.411806 28572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:59643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:45:38.431736 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44315
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:38.755441 28879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:38.755748 28879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:38.755903 28879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:38.763074 28879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:38.763190 28879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:38.763357 28879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:45:38.773078 28879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:38.774513 28879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:38.775921 28879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:38.793053 28885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:38.793260 28887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:38.793542 28884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:38.794484 28879 server_base.cc:1034] running on GCE node
I20260809 06:45:38.795217 28879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:38.796340 28879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:38.797681 28879 hybrid_clock.cc:648] HybridClock initialized: now 1786257938797600 us; error 65 us; skew 500 ppm
I20260809 06:45:38.798043 28879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:38 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257938, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:38.805580 28879 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:45:38.807121 28879 webserver.cc:460] Webserver started at http://127.27.209.194:32841/ using document root <none> and password file <none>
I20260809 06:45:38.807909 28879 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:38.808027 28879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:38.808338 28879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:38.811370 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "efd701bde37241fa8ff489dbd3bad4fb"
format_stamp: "Formatted at 2026-08-09 06:45:38 on dist-test-slave-jpj1"
server_key: "87cdd5157d42e87081261ccfaf6fb7c6"
server_key_iv: "5371aecf2c7c55edabb86c17e0d97884"
server_key_version: "encryptionkey@0"
I20260809 06:45:38.812197 28879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "efd701bde37241fa8ff489dbd3bad4fb"
format_stamp: "Formatted at 2026-08-09 06:45:38 on dist-test-slave-jpj1"
server_key: "87cdd5157d42e87081261ccfaf6fb7c6"
server_key_iv: "5371aecf2c7c55edabb86c17e0d97884"
server_key_version: "encryptionkey@0"
I20260809 06:45:38.817487 28879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:45:38.821118 28895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:38.822468 28879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:45:38.822657 28879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "efd701bde37241fa8ff489dbd3bad4fb"
format_stamp: "Formatted at 2026-08-09 06:45:38 on dist-test-slave-jpj1"
server_key: "87cdd5157d42e87081261ccfaf6fb7c6"
server_key_iv: "5371aecf2c7c55edabb86c17e0d97884"
server_key_version: "encryptionkey@0"
I20260809 06:45:38.822851 28879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:38.839756 28879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:38.843752 28879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:38.844018 28879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:38.844980 28879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:38.846297 28879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:38.846371 28879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:38.846486 28879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:38.846536 28879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:38.885553 28879 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:35689
I20260809 06:45:38.885587 29078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:35689 every 8 connection(s)
I20260809 06:45:38.886834 28879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:45:38.888262 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28879
I20260809 06:45:38.888442 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:45:38.888914 28487 external_mini_cluster.cc:1428] Setting key ade7ff3f5768c25aab0c36e585459dec
Aug 09 06:45:38 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257938, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
I20260809 06:45:38.907932 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:38.889474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:39023 (local address 127.27.209.254:44315)
0809 06:45:38.889693 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:38.889698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:38.891270 (+ 1572us) server_negotiation.cc:407] Connection header received
0809 06:45:38.893956 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:38.893961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:38.894031 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:38.894119 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:38.896118 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.896818 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.898497 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.898960 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.899057 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:38.902372 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:38.902377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:38.902381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:38.902414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:38.904985 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:38.905680 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:38.905683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:38.905686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:38.905811 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:38.906230 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:38.906233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:38.906235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:38.906496 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:38.906627 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:38.907330 (+ 703us) server_negotiation.cc:299] Negotiation successful
0809 06:45:38.907586 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:45:38.908921 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:38.890223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44315 (local address 127.27.209.194:39023)
0809 06:45:38.891110 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:38.891158 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:38.893635 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:38.894402 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:38.894415 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:38.895132 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:38.895939 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:38.895964 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.896968 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.896973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:38.898349 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:38.898362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:38.899158 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:38.899164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:38.899371 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:38.900020 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:38.900042 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:38.902150 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:38.905119 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:38.905129 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:38.905146 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:38.905556 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:38.905916 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:38.905921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:38.905925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:38.906112 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:38.906620 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:38.906627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:38.906962 (+ 335us) client_negotiation.cc:769] Sending connection context
0809 06:45:38.907212 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 06:45:38.907768 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":358,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
I20260809 06:45:38.910537 29080 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44315
I20260809 06:45:38.910903 29080 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:38.911896 29080 heartbeater.cc:511] Master 127.27.209.254:44315 requested a full tablet report, sending...
I20260809 06:45:38.913527 28572 ts_manager.cc:194] Registered new tserver with Master: efd701bde37241fa8ff489dbd3bad4fb (127.27.209.194:35689)
I20260809 06:45:38.914319 28572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:39023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:45:38.934289 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44315
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:39.262830 29087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:39.263121 29087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:39.263248 29087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:39.270678 29087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:39.270798 29087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:39.270957 29087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:45:39.280738 29087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44315
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:39.282053 29087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:39.283449 29087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:39.296311 29095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:39.296388 29097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:39.296346 29102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:39.297178 29087 server_base.cc:1034] running on GCE node
I20260809 06:45:39.297816 29087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:39.298707 29087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:39.299906 29087 hybrid_clock.cc:648] HybridClock initialized: now 1786257939299859 us; error 36 us; skew 500 ppm
I20260809 06:45:39.300215 29087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:39.303220 29087 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:45:39.304698 29087 webserver.cc:460] Webserver started at http://127.27.209.195:34267/ using document root <none> and password file <none>
I20260809 06:45:39.305389 29087 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:39.305502 29087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:39.305811 29087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:39.308691 29087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "8f52b6a40e7e48f4bcf1186fcb17d27c"
format_stamp: "Formatted at 2026-08-09 06:45:39 on dist-test-slave-jpj1"
server_key: "cdce5f99b2dedf33b09259da28de5d91"
server_key_iv: "f786741fa105d5aa9061a9611b014942"
server_key_version: "encryptionkey@0"
I20260809 06:45:39.309449 29087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "8f52b6a40e7e48f4bcf1186fcb17d27c"
format_stamp: "Formatted at 2026-08-09 06:45:39 on dist-test-slave-jpj1"
server_key: "cdce5f99b2dedf33b09259da28de5d91"
server_key_iv: "f786741fa105d5aa9061a9611b014942"
server_key_version: "encryptionkey@0"
I20260809 06:45:39.314683 29087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:45:39.318189 29112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:39.319512 29087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:45:39.319718 29087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8f52b6a40e7e48f4bcf1186fcb17d27c"
format_stamp: "Formatted at 2026-08-09 06:45:39 on dist-test-slave-jpj1"
server_key: "cdce5f99b2dedf33b09259da28de5d91"
server_key_iv: "f786741fa105d5aa9061a9611b014942"
server_key_version: "encryptionkey@0"
I20260809 06:45:39.319891 29087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:39.334323 29087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:39.338483 29087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:39.338734 29087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:39.339737 29087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:39.341063 29087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:39.341135 29087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:39.341229 29087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:39.341275 29087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:39.372997 29087 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:42709
I20260809 06:45:39.373042 29296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:42709 every 8 connection(s)
I20260809 06:45:39.374279 29087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:45:39.381882 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29087
I20260809 06:45:39.382091 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:45:39.382635 28487 external_mini_cluster.cc:1428] Setting key e7e475b398f4f5199ab873f002f477bb
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:39.394413 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.376952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:34057 (local address 127.27.209.254:44315)
0809 06:45:39.377153 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:39.377157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:39.378486 (+ 1329us) server_negotiation.cc:407] Connection header received
0809 06:45:39.380416 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:39.380423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:39.380501 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:39.380591 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:39.382183 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.383007 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.385077 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.385507 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.385618 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.389130 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:39.389136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:39.389139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:39.389173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:39.391632 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.392367 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.392371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.392373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.392512 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.392909 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.392912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.392914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.393183 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:39.393305 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:39.393979 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 06:45:39.394153 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:45:39.395397 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.377569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44315 (local address 127.27.209.195:34057)
0809 06:45:39.378311 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:39.378356 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:39.380132 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:39.380842 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:39.380850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:39.381380 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:39.381971 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:39.381996 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.383142 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.383147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:39.384940 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:39.384956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.385723 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.385731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:39.385968 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.386637 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:39.386666 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:39.388889 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:39.391791 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:39.391799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:39.391812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:39.392242 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:39.392610 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:39.392614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:39.392616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:39.392820 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:39.393309 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:39.393316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:39.393648 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 06:45:39.393892 (+ 244us) client_negotiation.cc:240] Negotiation successful
0809 06:45:39.394307 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":277,"threads_started":1}
I20260809 06:45:39.396873 29297 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44315
I20260809 06:45:39.397222 29297 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:39.398032 29297 heartbeater.cc:511] Master 127.27.209.254:44315 requested a full tablet report, sending...
I20260809 06:45:39.399487 28572 ts_manager.cc:194] Registered new tserver with Master: 8f52b6a40e7e48f4bcf1186fcb17d27c (127.27.209.195:42709)
I20260809 06:45:39.400362 28572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:34057
I20260809 06:45:39.414710 28865 heartbeater.cc:503] Master 127.27.209.254:44315 was elected leader, sending a full tablet report...
I20260809 06:45:39.418474 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:45:39.464828 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.449962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.27.209.193:34531)
0809 06:45:39.450642 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:39.450652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:39.450759 (+ 107us) server_negotiation.cc:407] Connection header received
0809 06:45:39.450962 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:39.450972 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:39.451189 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:39.451350 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:39.452791 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.453819 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.455063 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.455686 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.455808 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.459448 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:39.459458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:39.459472 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:39.459514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:39.462117 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.462745 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.462753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.462755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.462886 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.463263 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.463269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.463272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.463653 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:39.463946 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:39.464226 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 06:45:39.464486 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
W20260809 06:45:39.466022 28736 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54762
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:45:39.507790 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.495044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.27.209.193:34531)
0809 06:45:39.495272 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:39.495279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:39.495460 (+ 181us) server_negotiation.cc:407] Connection header received
0809 06:45:39.495696 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:39.495700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:39.495766 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:39.495848 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:39.497176 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.497885 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.499080 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.499500 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.499655 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.503311 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:39.503316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:39.503320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:39.503351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:39.505675 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.506226 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.506229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.506232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.506363 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.506664 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.506668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.506670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.506907 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:39.507084 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:39.507283 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 06:45:39.507442 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 06:45:39.509256 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28669
I20260809 06:45:39.521270 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.510259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.27.209.193:34531)
0809 06:45:39.510459 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:39.510464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:39.510655 (+ 191us) server_negotiation.cc:407] Connection header received
0809 06:45:39.510790 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:39.510795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:39.510853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:39.510927 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:39.512137 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.512819 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.514051 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.514471 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.514575 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.516669 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:39.516674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:39.516677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:39.516704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:39.518867 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.519458 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.519463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.519467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.519652 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.519986 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.519991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.519995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.520238 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:39.520461 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:39.520755 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 06:45:39.520947 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:45:39.521937 28858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:39.610121 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28669
I20260809 06:45:39.630050 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28879
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:45:39.644439 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.630879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.27.209.194:35689)
0809 06:45:39.631412 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:39.631419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:39.631445 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:45:39.631512 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:39.631518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:39.631800 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:39.631933 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:39.633006 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.633988 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.635107 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.635725 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.635817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.639477 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:39.639488 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:39.639502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:39.639561 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:39.642033 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.642633 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.642640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.642643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.642785 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.643102 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.643105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.643107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.643414 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:39.643784 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:39.643909 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 06:45:39.644096 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":280,"threads_started":1}
I20260809 06:45:39.645674 29070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:39.725251 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28879
I20260809 06:45:39.747268 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29087
Aug 09 06:45:39 dist-test-slave-jpj1 krb5kdc[28498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:45:39.763109 29325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:39.748250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.27.209.195:42709)
0809 06:45:39.748868 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:39.748876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:39.748905 (+ 29us) server_negotiation.cc:407] Connection header received
0809 06:45:39.748966 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:39.748973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:39.749219 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:39.749386 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:39.750778 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.751940 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.753201 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:39.753745 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:39.753858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:39.757783 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:39.757793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:39.757807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:39.757839 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:39.760499 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.761134 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.761141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.761143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.761270 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:39.761607 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:39.761611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:39.761613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:39.761941 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:39.762245 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:39.762525 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 06:45:39.762787 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":256,"threads_started":1}
I20260809 06:45:39.764456 29290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:39.844708 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29087
I20260809 06:45:39.864578 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28520
I20260809 06:45:39.865804 28625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:39.966056 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28520
2026-08-09T06:45:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4001 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:45:40 dist-test-slave-jpj1 krb5kdc[29335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:45:40 dist-test-slave-jpj1 krb5kdc[29335](info): set up 2 sockets
Aug 09 06:45:40 dist-test-slave-jpj1 krb5kdc[29335](info): commencing operation
krb5kdc: starting...
W20260809 06:45:41.148331 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:45:41 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:45:41.323199 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:45675
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:41.638772 29357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:41.639029 29357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:41.639154 29357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:41.646128 29357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:45:41.646224 29357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:41.646297 29357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:41.646369 29357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:45:41.646432 29357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:45:41.656023 29357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:41.657202 29357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:41.658505 29357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:41.669292 29368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:41.669474 29363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:41.669354 29366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:41.670395 29357 server_base.cc:1034] running on GCE node
I20260809 06:45:41.671133 29357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:41.672464 29357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:41.673666 29357 hybrid_clock.cc:648] HybridClock initialized: now 1786257941673610 us; error 46 us; skew 500 ppm
I20260809 06:45:41.673995 29357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:41 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:41.676894 29357 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:45:41.678261 29357 webserver.cc:460] Webserver started at http://127.27.209.254:39425/ using document root <none> and password file <none>
I20260809 06:45:41.678900 29357 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:41.679018 29357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:41.679329 29357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:41.682175 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "792257ec43a044dc989531e2ac21e609"
format_stamp: "Formatted at 2026-08-09 06:45:41 on dist-test-slave-jpj1"
server_key: "b53a8fb15697ec744e309312a5df529d"
server_key_iv: "ca38b2130887dd2a0976ace1be805749"
server_key_version: "encryptionkey@0"
I20260809 06:45:41.682888 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "792257ec43a044dc989531e2ac21e609"
format_stamp: "Formatted at 2026-08-09 06:45:41 on dist-test-slave-jpj1"
server_key: "b53a8fb15697ec744e309312a5df529d"
server_key_iv: "ca38b2130887dd2a0976ace1be805749"
server_key_version: "encryptionkey@0"
I20260809 06:45:41.687762 29357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 06:45:41.690945 29376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:41.692169 29357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:45:41.692338 29357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "792257ec43a044dc989531e2ac21e609"
format_stamp: "Formatted at 2026-08-09 06:45:41 on dist-test-slave-jpj1"
server_key: "b53a8fb15697ec744e309312a5df529d"
server_key_iv: "ca38b2130887dd2a0976ace1be805749"
server_key_version: "encryptionkey@0"
I20260809 06:45:41.692508 29357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:41.707937 29357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:41.711920 29357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:41.712167 29357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:41.729017 29357 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:45675
I20260809 06:45:41.729054 29458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:45675 every 8 connection(s)
I20260809 06:45:41.730341 29357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:45:41.734050 29459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:41.739187 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29357
I20260809 06:45:41.739426 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:45:41.740227 28487 external_mini_cluster.cc:1428] Setting key 9f10a59b7cbdc65e641ab9388ff578b7
I20260809 06:45:41.745240 29459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609: Bootstrap starting.
I20260809 06:45:41.748811 29459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:41.750059 29459 log.cc:824] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:41.752938 29459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609: No bootstrap required, opened a new log
Aug 09 06:45:41 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:41.760489 29459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792257ec43a044dc989531e2ac21e609" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 45675 } }
I20260809 06:45:41.760869 29459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:41.760984 29459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792257ec43a044dc989531e2ac21e609, State: Initialized, Role: FOLLOWER
I20260809 06:45:41.761565 29459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792257ec43a044dc989531e2ac21e609" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 45675 } }
I20260809 06:45:41.761806 29459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:45:41.761931 29459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:45:41.762049 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:41.742260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.27.209.254:45675)
0809 06:45:41.743070 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:41.743082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:41.743124 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:45:41.744849 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:41.744873 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:41.745260 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:41.745582 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:41.747365 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:41.748688 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:41.750390 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:41.751129 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:41.751315 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:41.755187 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:41.755195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:41.755209 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:41.755242 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:41.758180 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:41.758797 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:41.758804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:41.758811 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:41.758979 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:41.759321 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:41.759325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:41.759327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:41.759874 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:41.760147 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:41.760627 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 06:45:41.761042 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":617,"thread_start_us":338,"threads_started":1}
I20260809 06:45:41.762100 29459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:41.764384 29459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792257ec43a044dc989531e2ac21e609" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 45675 } }
I20260809 06:45:41.764853 29459 leader_election.cc:302] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792257ec43a044dc989531e2ac21e609; no voters:
I20260809 06:45:41.765358 29459 leader_election.cc:288] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:45:41.765456 29467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:41.766088 29467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [term 1 LEADER]: Becoming Leader. State: Replica: 792257ec43a044dc989531e2ac21e609, State: Running, Role: LEADER
I20260809 06:45:41.766566 29467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792257ec43a044dc989531e2ac21e609" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 45675 } }
I20260809 06:45:41.767244 29459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:45:41.768975 29468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792257ec43a044dc989531e2ac21e609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792257ec43a044dc989531e2ac21e609" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 45675 } } }
I20260809 06:45:41.769313 29468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:41.770081 29472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:45:41.769964 29469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792257ec43a044dc989531e2ac21e609. Latest consensus state: current_term: 1 leader_uuid: "792257ec43a044dc989531e2ac21e609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792257ec43a044dc989531e2ac21e609" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 45675 } } }
I20260809 06:45:41.770989 29469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609 [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:41.776322 29472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:45:41.784787 29472 catalog_manager.cc:1347] Generated new cluster ID: 7758bfc670ec4f49937e77e8aa3a2424
I20260809 06:45:41.784895 29472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:45:41.825893 29472 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:45:41.827271 29472 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:45:41.838089 29472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792257ec43a044dc989531e2ac21e609: Generated new TSK 0
I20260809 06:45:41.838867 29472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:45:41.895824 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:45675
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:42.220935 29499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:42.221246 29499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:42.221385 29499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:42.228519 29499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:42.228639 29499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:42.228801 29499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:45:42.238550 29499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:45675
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:42.239872 29499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:42.241189 29499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:42.253799 29507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:42.253702 29508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:42.253842 29511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:42.254879 29499 server_base.cc:1034] running on GCE node
I20260809 06:45:42.255471 29499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:42.256394 29499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:42.257593 29499 hybrid_clock.cc:648] HybridClock initialized: now 1786257942257541 us; error 40 us; skew 500 ppm
I20260809 06:45:42.257894 29499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:42 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:42.260926 29499 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:45:42.262544 29499 webserver.cc:460] Webserver started at http://127.27.209.193:34793/ using document root <none> and password file <none>
I20260809 06:45:42.263236 29499 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:42.263356 29499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:42.263705 29499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:42.266764 29499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "bda2e5525c634c8fb9d1c353828506e5"
format_stamp: "Formatted at 2026-08-09 06:45:42 on dist-test-slave-jpj1"
server_key: "6ade5f2cbe988b2927d2d90134d0cdc7"
server_key_iv: "870ed3ef7b5ad866b0520f69f1c56d0d"
server_key_version: "encryptionkey@0"
I20260809 06:45:42.267594 29499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "bda2e5525c634c8fb9d1c353828506e5"
format_stamp: "Formatted at 2026-08-09 06:45:42 on dist-test-slave-jpj1"
server_key: "6ade5f2cbe988b2927d2d90134d0cdc7"
server_key_iv: "870ed3ef7b5ad866b0520f69f1c56d0d"
server_key_version: "encryptionkey@0"
I20260809 06:45:42.272692 29499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:45:42.276254 29526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:42.277571 29499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 06:45:42.277763 29499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "bda2e5525c634c8fb9d1c353828506e5"
format_stamp: "Formatted at 2026-08-09 06:45:42 on dist-test-slave-jpj1"
server_key: "6ade5f2cbe988b2927d2d90134d0cdc7"
server_key_iv: "870ed3ef7b5ad866b0520f69f1c56d0d"
server_key_version: "encryptionkey@0"
I20260809 06:45:42.278002 29499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:42.300233 29499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:42.303949 29499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:42.304190 29499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:42.305059 29499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:42.306277 29499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:42.306351 29499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:42.306440 29499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:42.306490 29499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:42.337462 29499 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:38683
I20260809 06:45:42.337522 29703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:38683 every 8 connection(s)
I20260809 06:45:42.338809 29499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:45:42.344383 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29499
I20260809 06:45:42.344571 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:45:42.345335 28487 external_mini_cluster.cc:1428] Setting key 40f4750694b2a1030df8f32b1efae7ed
Aug 09 06:45:42 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:42.360152 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:42.341325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:41789 (local address 127.27.209.254:45675)
0809 06:45:42.341740 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:42.341745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:42.342718 (+ 973us) server_negotiation.cc:407] Connection header received
0809 06:45:42.344747 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:42.344754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:42.344846 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:42.344975 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:42.346651 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.347674 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.349971 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.350446 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.350552 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:42.354611 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:42.354616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:42.354619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:42.354651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:42.357045 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:42.357877 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:42.357881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:42.357883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:42.358004 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:42.358475 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:42.358478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:42.358480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:42.358747 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:42.358857 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:42.359724 (+ 867us) server_negotiation.cc:299] Negotiation successful
0809 06:45:42.359896 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":195,"threads_started":1}
I20260809 06:45:42.361584 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:42.341877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:45675 (local address 127.27.209.193:41789)
0809 06:45:42.342566 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:42.342604 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:42.344331 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:42.345172 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:42.345181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:42.345781 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:42.346474 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:42.346497 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.347839 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.347845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:42.349800 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:42.349816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.350683 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.350692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:42.350979 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:42.351737 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:42.351766 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:42.354381 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:42.357193 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:42.357202 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:42.357219 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:42.357761 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:42.358113 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:42.358118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:42.358121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:42.358371 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:42.358874 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:42.358882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:42.359289 (+ 407us) client_negotiation.cc:769] Sending connection context
0809 06:45:42.359633 (+ 344us) client_negotiation.cc:240] Negotiation successful
0809 06:45:42.360159 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":250,"threads_started":1}
I20260809 06:45:42.363247 29704 heartbeater.cc:347] Connected to a master server at 127.27.209.254:45675
I20260809 06:45:42.363585 29704 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:42.364302 29704 heartbeater.cc:511] Master 127.27.209.254:45675 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
I20260809 06:45:42.366369 29405 ts_manager.cc:194] Registered new tserver with Master: bda2e5525c634c8fb9d1c353828506e5 (127.27.209.193:38683)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
I20260809 06:45:42.368043 29405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:41789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:45:42.390450 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:45675
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:42.709224 29712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:42.709517 29712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:42.709677 29712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:42.716595 29712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:42.716707 29712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:42.716857 29712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:45:42.726016 29712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:45675
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:42.727238 29712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:42.728605 29712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:42.741005 29724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:42.741206 29725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:42.741865 29727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:42.742249 29712 server_base.cc:1034] running on GCE node
I20260809 06:45:42.742919 29712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:42.744022 29712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:42.745236 29712 hybrid_clock.cc:648] HybridClock initialized: now 1786257942745180 us; error 50 us; skew 500 ppm
I20260809 06:45:42.745530 29712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:42 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:42.748641 29712 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:45:42.750031 29712 webserver.cc:460] Webserver started at http://127.27.209.194:35563/ using document root <none> and password file <none>
I20260809 06:45:42.750689 29712 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:42.750797 29712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:42.751101 29712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:42.753914 29712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "6c677101bc274f268453e746cf5ac5dc"
format_stamp: "Formatted at 2026-08-09 06:45:42 on dist-test-slave-jpj1"
server_key: "d0681ec62671eb0f0ed4343bb4f48289"
server_key_iv: "8dfd9a0bd06a41d593241e9e30df4926"
server_key_version: "encryptionkey@0"
I20260809 06:45:42.754662 29712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "6c677101bc274f268453e746cf5ac5dc"
format_stamp: "Formatted at 2026-08-09 06:45:42 on dist-test-slave-jpj1"
server_key: "d0681ec62671eb0f0ed4343bb4f48289"
server_key_iv: "8dfd9a0bd06a41d593241e9e30df4926"
server_key_version: "encryptionkey@0"
I20260809 06:45:42.759443 29712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 06:45:42.762779 29738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:42.764038 29712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:45:42.764202 29712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "6c677101bc274f268453e746cf5ac5dc"
format_stamp: "Formatted at 2026-08-09 06:45:42 on dist-test-slave-jpj1"
server_key: "d0681ec62671eb0f0ed4343bb4f48289"
server_key_iv: "8dfd9a0bd06a41d593241e9e30df4926"
server_key_version: "encryptionkey@0"
I20260809 06:45:42.764375 29712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:42.776973 29712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:42.780681 29712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:42.780958 29712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:42.781885 29712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:42.783191 29712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:42.783268 29712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:42.783387 29712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:42.783447 29712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:42.812572 29712 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:38669
I20260809 06:45:42.812616 29912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:38669 every 8 connection(s)
I20260809 06:45:42.814010 29712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:45:42.817534 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29712
I20260809 06:45:42.817740 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:45:42.818328 28487 external_mini_cluster.cc:1428] Setting key fa4234ec0c5bc12524fe1e119edea8a3
Aug 09 06:45:42 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:42.834416 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:42.816622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:53731 (local address 127.27.209.254:45675)
0809 06:45:42.816842 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:42.816848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:42.818122 (+ 1274us) server_negotiation.cc:407] Connection header received
0809 06:45:42.820029 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:42.820034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:42.820139 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:42.820224 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:42.821676 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.822384 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.823916 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.824335 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.824444 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:42.828228 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:42.828235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:42.828239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:42.828279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:42.831361 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:42.832073 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:42.832079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:42.832082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:42.832252 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:42.832719 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:42.832724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:42.832727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:42.833110 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:42.833313 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:42.833985 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 06:45:42.834155 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:45:42.835775 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:42.817121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:45675 (local address 127.27.209.194:53731)
0809 06:45:42.817972 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:42.818010 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:42.819779 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:42.820458 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:42.820466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:42.820978 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:42.821516 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:42.821533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.822522 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.822526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:42.823783 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:42.823795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:42.824540 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:42.824547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:42.824771 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:42.825426 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:42.825455 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:42.827999 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:42.831509 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:42.831515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:42.831526 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:42.831941 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:42.832359 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:42.832362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:42.832365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:42.832539 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:42.833261 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:42.833267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:42.833594 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 06:45:42.833888 (+ 294us) client_negotiation.cc:240] Negotiation successful
0809 06:45:42.834409 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":335,"threads_started":1}
I20260809 06:45:42.837137 29914 heartbeater.cc:347] Connected to a master server at 127.27.209.254:45675
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
I20260809 06:45:42.837474 29914 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:42.838254 29914 heartbeater.cc:511] Master 127.27.209.254:45675 requested a full tablet report, sending...
I20260809 06:45:42.839781 29405 ts_manager.cc:194] Registered new tserver with Master: 6c677101bc274f268453e746cf5ac5dc (127.27.209.194:38669)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
I20260809 06:45:42.840677 29405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:53731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:45:42.860224 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:45675
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:43.177872 29920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:43.178157 29920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:43.178259 29920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:43.185452 29920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:43.185563 29920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:43.185690 29920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:45:43.195210 29920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:45675
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:43.196596 29920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:43.197962 29920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:43.210217 29927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:43.210701 29931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:43.211031 29928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:43.211722 29920 server_base.cc:1034] running on GCE node
I20260809 06:45:43.212335 29920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:43.213292 29920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:43.214509 29920 hybrid_clock.cc:648] HybridClock initialized: now 1786257943214453 us; error 45 us; skew 500 ppm
I20260809 06:45:43.214843 29920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:43.217964 29920 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:45:43.219417 29920 webserver.cc:460] Webserver started at http://127.27.209.195:35217/ using document root <none> and password file <none>
I20260809 06:45:43.220110 29920 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:43.220232 29920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:43.220535 29920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:43.223337 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "8ffa3c1f1c884769af39893c151d1f65"
format_stamp: "Formatted at 2026-08-09 06:45:43 on dist-test-slave-jpj1"
server_key: "625718fd436e01af0aa9592690f41806"
server_key_iv: "740c64d61ea4656806a60dd179585226"
server_key_version: "encryptionkey@0"
I20260809 06:45:43.224148 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "8ffa3c1f1c884769af39893c151d1f65"
format_stamp: "Formatted at 2026-08-09 06:45:43 on dist-test-slave-jpj1"
server_key: "625718fd436e01af0aa9592690f41806"
server_key_iv: "740c64d61ea4656806a60dd179585226"
server_key_version: "encryptionkey@0"
I20260809 06:45:43.229044 29920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:45:43.232420 29947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:43.233728 29920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:45:43.233898 29920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8ffa3c1f1c884769af39893c151d1f65"
format_stamp: "Formatted at 2026-08-09 06:45:43 on dist-test-slave-jpj1"
server_key: "625718fd436e01af0aa9592690f41806"
server_key_iv: "740c64d61ea4656806a60dd179585226"
server_key_version: "encryptionkey@0"
I20260809 06:45:43.234081 29920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:43.260516 29920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:43.264690 29920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:43.264954 29920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:43.265900 29920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:43.267213 29920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:43.267306 29920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:43.267477 29920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:43.267545 29920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:43.298138 29920 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:37787
I20260809 06:45:43.298185 30120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:37787 every 8 connection(s)
I20260809 06:45:43.299364 29920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:45:43.307214 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29920
I20260809 06:45:43.307389 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:45:43.307976 28487 external_mini_cluster.cc:1428] Setting key 487d32d769442b852083730cbade322c
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:43.318821 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.301752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:39649 (local address 127.27.209.254:45675)
0809 06:45:43.301950 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.301955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.303141 (+ 1186us) server_negotiation.cc:407] Connection header received
0809 06:45:43.304910 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.304915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.304981 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.305067 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.306500 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.307241 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.309394 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.309906 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.310026 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.313705 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.313710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.313714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.313746 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.316286 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.316897 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.316901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.316903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.317016 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.317348 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.317351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.317353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.317598 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.317744 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.318363 (+ 619us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.318544 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:45:43.319630 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.302243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:45675 (local address 127.27.209.195:39649)
0809 06:45:43.302974 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:43.303019 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:43.304674 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:43.305254 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:43.305261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:43.305811 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:43.306365 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.306382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.307415 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.307420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.309258 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.309273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.310389 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.310398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.310691 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.311348 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:43.311370 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:43.313503 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:43.316400 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:43.316406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:43.316417 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:43.316802 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:43.317091 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:43.317094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:43.317096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:43.317261 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:43.317730 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:43.317736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:43.318017 (+ 281us) client_negotiation.cc:769] Sending connection context
0809 06:45:43.318226 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 06:45:43.318592 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":293,"threads_started":1}
I20260809 06:45:43.320863 30121 heartbeater.cc:347] Connected to a master server at 127.27.209.254:45675
I20260809 06:45:43.321117 30121 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:43.321760 30121 heartbeater.cc:511] Master 127.27.209.254:45675 requested a full tablet report, sending...
I20260809 06:45:43.323132 29398 ts_manager.cc:194] Registered new tserver with Master: 8ffa3c1f1c884769af39893c151d1f65 (127.27.209.195:37787)
I20260809 06:45:43.324007 29398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:39649
I20260809 06:45:43.335088 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:45:43.354930 29707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.342260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.27.209.254:45675)
0809 06:45:43.342444 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.342449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.343573 (+ 1124us) server_negotiation.cc:407] Connection header received
0809 06:45:43.343747 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.343752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.343813 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.343894 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.345336 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.346067 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.347328 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.347768 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.347878 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.350071 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.350075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.350078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.350108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.352457 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.353063 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.353067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.353069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.353196 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.353533 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.353539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.353542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.353865 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.354112 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.354428 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.354642 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:45:43.360961 29398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:45:43.367797 29398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:43.371174 29704 heartbeater.cc:503] Master 127.27.209.254:45675 was elected leader, sending a full tablet report...
I20260809 06:45:43.389294 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.380088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.27.209.193:38683)
0809 06:45:43.381597 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.381606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.381635 (+ 29us) server_negotiation.cc:407] Connection header received
0809 06:45:43.381762 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.381770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.382006 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.382212 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:43.383077 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.384693 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.386682 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.388018 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.388215 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.388389 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.388797 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.388959 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":1344,"threads_started":1}
I20260809 06:45:43.389317 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.379456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38683 (local address 127.0.0.1:57816)
0809 06:45:43.380844 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:43.380861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:43.380983 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:43.382477 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:43.382484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:43.382513 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:43.382888 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.382901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.384826 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.384831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.386526 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.386539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.388300 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.388307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.388519 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.388574 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 06:45:43.388712 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 06:45:43.388887 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"thread_start_us":1119,"threads_started":1}
I20260809 06:45:43.390180 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.381386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38669 (local address 127.0.0.1:55814)
0809 06:45:43.381940 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:43.381956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:43.382055 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:43.383085 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:43.383089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:43.383113 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:43.383480 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.383493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.385212 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.385217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.387324 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.387337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.389447 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.389455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.389625 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.389660 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:45:43.389743 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 06:45:43.389866 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":328,"threads_started":1}
I20260809 06:45:43.390451 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.381600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.27.209.194:38669)
0809 06:45:43.382526 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.382534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.382561 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:45:43.382686 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.382695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.382927 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.383103 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:43.383739 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.385089 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.387644 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.388962 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.389134 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.389288 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.389979 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.390151 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":667,"threads_started":1}
I20260809 06:45:43.391939 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.379332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37787 (local address 127.0.0.1:55810)
0809 06:45:43.380027 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:43.380078 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:43.380248 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:43.384625 (+ 4377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:43.384630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:43.384650 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:43.385062 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.385074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.387029 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.387034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.389480 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.389497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.391187 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.391194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.391361 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.391392 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:45:43.391472 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 06:45:43.391623 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
I20260809 06:45:43.392086 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.379658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.27.209.195:37787)
0809 06:45:43.383792 (+ 4134us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.383800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.383838 (+ 38us) server_negotiation.cc:407] Connection header received
0809 06:45:43.383930 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.383937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.384147 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.384340 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:43.385278 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.386879 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.389684 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.391000 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.391176 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.391339 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.391522 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.391752 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3972,"thread_start_us":3785,"threads_started":1}
I20260809 06:45:43.393985 29604 tablet_service.cc:1467] Processing CreateTablet for tablet 4511af5c3bc34e419aa0ca07394d856f (DEFAULT_TABLE table=test-table [id=16f34f3bea424bb0823bb9d059c92122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:43.394513 29808 tablet_service.cc:1467] Processing CreateTablet for tablet 4511af5c3bc34e419aa0ca07394d856f (DEFAULT_TABLE table=test-table [id=16f34f3bea424bb0823bb9d059c92122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:43.395157 30020 tablet_service.cc:1467] Processing CreateTablet for tablet 4511af5c3bc34e419aa0ca07394d856f (DEFAULT_TABLE table=test-table [id=16f34f3bea424bb0823bb9d059c92122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:43.395619 29604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4511af5c3bc34e419aa0ca07394d856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:43.395947 29808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4511af5c3bc34e419aa0ca07394d856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:43.396325 30020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4511af5c3bc34e419aa0ca07394d856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:43.411800 30154 tablet_bootstrap.cc:492] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: Bootstrap starting.
I20260809 06:45:43.415920 30154 tablet_bootstrap.cc:654] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:43.416316 30155 tablet_bootstrap.cc:492] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: Bootstrap starting.
I20260809 06:45:43.417526 30154 log.cc:824] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:43.420179 30155 tablet_bootstrap.cc:654] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:43.420410 30154 tablet_bootstrap.cc:492] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: No bootstrap required, opened a new log
I20260809 06:45:43.420660 30154 ts_tablet_manager.cc:1397] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 06:45:43.421483 30155 log.cc:824] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:43.423121 30156 tablet_bootstrap.cc:492] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: Bootstrap starting.
I20260809 06:45:43.426204 30155 tablet_bootstrap.cc:492] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: No bootstrap required, opened a new log
I20260809 06:45:43.426414 30156 tablet_bootstrap.cc:654] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:43.426483 30155 ts_tablet_manager.cc:1397] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 06:45:43.427690 30156 log.cc:824] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:43.428570 30154 raft_consensus.cc:348] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.429021 30154 raft_consensus.cc:374] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:43.429127 30154 raft_consensus.cc:729] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffa3c1f1c884769af39893c151d1f65, State: Initialized, Role: FOLLOWER
I20260809 06:45:43.429718 30154 consensus_queue.cc:260] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.430744 30156 tablet_bootstrap.cc:492] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: No bootstrap required, opened a new log
I20260809 06:45:43.430995 30156 ts_tablet_manager.cc:1397] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:45:43.436215 30121 heartbeater.cc:503] Master 127.27.209.254:45675 was elected leader, sending a full tablet report...
I20260809 06:45:43.436236 30155 raft_consensus.cc:348] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.436635 30155 raft_consensus.cc:374] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:43.436738 30155 raft_consensus.cc:729] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda2e5525c634c8fb9d1c353828506e5, State: Initialized, Role: FOLLOWER
I20260809 06:45:43.437312 30155 consensus_queue.cc:260] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.438184 30154 ts_tablet_manager.cc:1428] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: Time spent starting tablet: real 0.017s user 0.006s sys 0.010s
I20260809 06:45:43.441282 30155 ts_tablet_manager.cc:1428] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260809 06:45:43.443105 30156 raft_consensus.cc:348] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.443490 30156 raft_consensus.cc:374] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:43.443608 30156 raft_consensus.cc:729] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c677101bc274f268453e746cf5ac5dc, State: Initialized, Role: FOLLOWER
I20260809 06:45:43.444159 30156 consensus_queue.cc:260] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.445230 29914 heartbeater.cc:503] Master 127.27.209.254:45675 was elected leader, sending a full tablet report...
I20260809 06:45:43.445624 30156 ts_tablet_manager.cc:1428] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260809 06:45:43.496822 30167 raft_consensus.cc:482] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:43.497109 30167 raft_consensus.cc:504] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.498745 30167 leader_election.cc:288] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ffa3c1f1c884769af39893c151d1f65 (127.27.209.195:37787), 6c677101bc274f268453e746cf5ac5dc (127.27.209.194:38669)
I20260809 06:45:43.504981 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.499098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:34607 (local address 127.27.209.195:37787)
0809 06:45:43.499281 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.499292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.499864 (+ 572us) server_negotiation.cc:407] Connection header received
0809 06:45:43.500086 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.500093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.500172 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.500278 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:43.500844 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.501566 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.503242 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.504045 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.504188 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.504263 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.504593 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.504733 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:45:43.505013 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.499032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37787 (local address 127.27.209.193:34607)
0809 06:45:43.499765 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:43.499786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:43.499929 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:43.500327 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:43.500332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:43.500359 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:43.500701 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.500711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.501688 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.501692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.503098 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.503110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.504250 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.504255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.504412 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.504443 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:45:43.504540 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 06:45:43.504661 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":360,"threads_started":1}
I20260809 06:45:43.505689 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.500134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:50523 (local address 127.27.209.194:38669)
0809 06:45:43.500341 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.500347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.500578 (+ 231us) server_negotiation.cc:407] Connection header received
0809 06:45:43.500757 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.500762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.500816 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.500888 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:43.501407 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.502060 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.503717 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.504595 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.504734 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.504798 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.505327 (+ 529us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.505452 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:45:43.506186 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.499887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38669 (local address 127.27.209.193:50523)
0809 06:45:43.500479 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:43.500495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:43.500622 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:43.500929 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:43.500933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:43.500956 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:43.501264 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.501273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.502200 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.502207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.503567 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:43.503579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.504954 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.504960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:43.505145 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.505184 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 06:45:43.505855 (+ 671us) client_negotiation.cc:240] Negotiation successful
0809 06:45:43.505951 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":457,"threads_started":1}
I20260809 06:45:43.506121 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4511af5c3bc34e419aa0ca07394d856f" candidate_uuid: "bda2e5525c634c8fb9d1c353828506e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffa3c1f1c884769af39893c151d1f65" is_pre_election: true
I20260809 06:45:43.506628 30044 raft_consensus.cc:2454] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bda2e5525c634c8fb9d1c353828506e5 in term 0.
I20260809 06:45:43.507115 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4511af5c3bc34e419aa0ca07394d856f" candidate_uuid: "bda2e5525c634c8fb9d1c353828506e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c677101bc274f268453e746cf5ac5dc" is_pre_election: true
I20260809 06:45:43.507314 29527 leader_election.cc:302] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ffa3c1f1c884769af39893c151d1f65, bda2e5525c634c8fb9d1c353828506e5; no voters:
I20260809 06:45:43.507592 29835 raft_consensus.cc:2454] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bda2e5525c634c8fb9d1c353828506e5 in term 0.
I20260809 06:45:43.507709 30167 raft_consensus.cc:2781] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:43.507865 30167 raft_consensus.cc:482] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:43.507968 30167 raft_consensus.cc:3037] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:43.510106 30167 raft_consensus.cc:504] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.510627 30167 leader_election.cc:288] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [CANDIDATE]: Term 1 election: Requested vote from peers 8ffa3c1f1c884769af39893c151d1f65 (127.27.209.195:37787), 6c677101bc274f268453e746cf5ac5dc (127.27.209.194:38669)
I20260809 06:45:43.511035 30044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4511af5c3bc34e419aa0ca07394d856f" candidate_uuid: "bda2e5525c634c8fb9d1c353828506e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffa3c1f1c884769af39893c151d1f65"
I20260809 06:45:43.511135 29835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4511af5c3bc34e419aa0ca07394d856f" candidate_uuid: "bda2e5525c634c8fb9d1c353828506e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c677101bc274f268453e746cf5ac5dc"
I20260809 06:45:43.511214 30044 raft_consensus.cc:3037] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:43.511353 29835 raft_consensus.cc:3037] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:43.513357 30044 raft_consensus.cc:2454] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bda2e5525c634c8fb9d1c353828506e5 in term 1.
I20260809 06:45:43.513571 29835 raft_consensus.cc:2454] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bda2e5525c634c8fb9d1c353828506e5 in term 1.
I20260809 06:45:43.513689 29527 leader_election.cc:302] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ffa3c1f1c884769af39893c151d1f65, bda2e5525c634c8fb9d1c353828506e5; no voters:
I20260809 06:45:43.513991 30167 raft_consensus.cc:2781] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:43.514405 30167 raft_consensus.cc:686] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [term 1 LEADER]: Becoming Leader. State: Replica: bda2e5525c634c8fb9d1c353828506e5, State: Running, Role: LEADER
I20260809 06:45:43.514910 30167 consensus_queue.cc:237] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } }
I20260809 06:45:43.519361 29405 catalog_manager.cc:5495] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to bda2e5525c634c8fb9d1c353828506e5 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "bda2e5525c634c8fb9d1c353828506e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa3c1f1c884769af39893c151d1f65" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bda2e5525c634c8fb9d1c353828506e5" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c677101bc274f268453e746cf5ac5dc" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38669 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
W20260809 06:45:43.550978 30122 tablet.cc:2369] T 4511af5c3bc34e419aa0ca07394d856f P 8ffa3c1f1c884769af39893c151d1f65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:43.551071 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.537113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.27.209.193:38683)
0809 06:45:43.537341 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.537348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.537711 (+ 363us) server_negotiation.cc:407] Connection header received
0809 06:45:43.537880 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.537885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.537946 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.538028 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.539563 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.540272 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.541500 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.541910 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.542019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.545794 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.545807 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.545824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.545863 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.548436 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.549066 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.549074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.549076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.549245 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.549600 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.549603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.549605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.549934 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.550222 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.550490 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.550762 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 06:45:43.565953 29915 tablet.cc:2369] T 4511af5c3bc34e419aa0ca07394d856f P 6c677101bc274f268453e746cf5ac5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
W20260809 06:45:43.591005 29705 tablet.cc:2369] T 4511af5c3bc34e419aa0ca07394d856f P bda2e5525c634c8fb9d1c353828506e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:43.595858 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.582751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.27.209.193:38683)
0809 06:45:43.582975 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.582982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.583210 (+ 228us) server_negotiation.cc:407] Connection header received
0809 06:45:43.583400 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.583404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.583469 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.583596 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.584931 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.585629 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.586904 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.587327 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.587441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.591028 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.591033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.591036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.591068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.593402 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.593968 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.593972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.593974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.594103 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.594482 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.594486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.594488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.594753 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.595027 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.595252 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.595467 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:45:43.639684 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.627033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.27.209.193:38683)
0809 06:45:43.627262 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.627269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.627480 (+ 211us) server_negotiation.cc:407] Connection header received
0809 06:45:43.627665 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.627670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.627754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.627839 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.629114 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.629866 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.630970 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.631353 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.631441 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.635018 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.635024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.635027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.635058 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.637403 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.638011 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.638015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.638017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.638148 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.638478 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.638481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.638483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.638765 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.638970 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.639148 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.639299 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 06:45:43.641508 29571 tablet_service.cc:2908] Rejecting scan request for tablet 4511af5c3bc34e419aa0ca07394d856f: Uninitialized: clean time has not yet been initialized
I20260809 06:45:43.644474 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29499
I20260809 06:45:43.656442 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.645623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.27.209.193:38683)
0809 06:45:43.645756 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.645760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.646137 (+ 377us) server_negotiation.cc:407] Connection header received
0809 06:45:43.646279 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.646282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.646343 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.646415 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.647484 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.648192 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.649355 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.649751 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.649835 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.651899 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.651904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.651907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.651933 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.654160 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.654725 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.654728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.654730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.654827 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.655157 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.655159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.655161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.655398 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.655605 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.655917 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.656153 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 06:45:43.657007 29696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:43.744561 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29499
I20260809 06:45:43.767625 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29712
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:45:43.781491 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.768357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.27.209.194:38669)
0809 06:45:43.768525 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.768532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.768612 (+ 80us) server_negotiation.cc:407] Connection header received
0809 06:45:43.768800 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.768804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.768878 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.768956 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.770132 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.770890 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.772226 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.772749 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.772848 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.776529 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.776541 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.776556 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.776591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.779135 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.779810 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.779818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.779821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.779947 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.780223 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.780226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.780228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.780588 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.780838 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.781003 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.781166 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 06:45:43.782092 29903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:43.866449 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29712
I20260809 06:45:43.894095 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29920
Aug 09 06:45:43 dist-test-slave-jpj1 krb5kdc[29335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:45:43.911661 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:43.894901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.27.209.195:37787)
0809 06:45:43.895087 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:43.895093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:43.895114 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:45:43.895284 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:43.895288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:43.895367 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:43.895456 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:43.896735 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.897511 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.898795 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:43.899192 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:43.899305 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:43.906257 (+ 6952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:43.906275 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:43.906293 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:43.906328 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:43.908986 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.909629 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.909637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.909639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.909792 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:43.910150 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:43.910154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:43.910156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:43.910513 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:43.910800 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:43.911096 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 06:45:43.911313 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:45:43.912300 30111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:43.997707 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29920
I20260809 06:45:44.021347 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29357
I20260809 06:45:44.022576 29449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:44.130586 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29357
2026-08-09T06:45:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4164 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:45:44 dist-test-slave-jpj1 krb5kdc[30197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:45:44 dist-test-slave-jpj1 krb5kdc[30197](info): set up 2 sockets
Aug 09 06:45:44 dist-test-slave-jpj1 krb5kdc[30197](info): commencing operation
krb5kdc: starting...
W20260809 06:45:45.293180 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:45:45 dist-test-slave-jpj1 krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:45:45.474776 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40379
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:45.791886 30214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:45.792160 30214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:45.792289 30214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:45.799360 30214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:45:45.799474 30214 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 06:45:45.799556 30214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:45.799647 30214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:45.799714 30214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:45:45.799803 30214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:45:45.809836 30214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:40379
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:45.811142 30214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:45.812474 30214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:45.823871 30223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:45.823844 30226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:45.823827 30224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:45.824694 30214 server_base.cc:1034] running on GCE node
I20260809 06:45:45.825438 30214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:45.826788 30214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:45.828006 30214 hybrid_clock.cc:648] HybridClock initialized: now 1786257945827931 us; error 40 us; skew 500 ppm
I20260809 06:45:45.828305 30214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:45 dist-test-slave-jpj1 krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257945, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:45.831341 30214 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:45:45.832782 30214 webserver.cc:460] Webserver started at http://127.27.209.254:33339/ using document root <none> and password file <none>
I20260809 06:45:45.833427 30214 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:45.833513 30214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:45.833792 30214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:45.836635 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "88cb5b530ecf4abbaaaa53dc17c0177f"
format_stamp: "Formatted at 2026-08-09 06:45:45 on dist-test-slave-jpj1"
server_key: "2d8f4cd83a58d9d40569de1b2c9118e8"
server_key_iv: "f767b52e17e2fec9cd1d5d1f069c7128"
server_key_version: "encryptionkey@0"
I20260809 06:45:45.837345 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "88cb5b530ecf4abbaaaa53dc17c0177f"
format_stamp: "Formatted at 2026-08-09 06:45:45 on dist-test-slave-jpj1"
server_key: "2d8f4cd83a58d9d40569de1b2c9118e8"
server_key_iv: "f767b52e17e2fec9cd1d5d1f069c7128"
server_key_version: "encryptionkey@0"
I20260809 06:45:45.842219 30214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:45:45.845485 30235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:45.846886 30214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:45:45.847050 30214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "88cb5b530ecf4abbaaaa53dc17c0177f"
format_stamp: "Formatted at 2026-08-09 06:45:45 on dist-test-slave-jpj1"
server_key: "2d8f4cd83a58d9d40569de1b2c9118e8"
server_key_iv: "f767b52e17e2fec9cd1d5d1f069c7128"
server_key_version: "encryptionkey@0"
I20260809 06:45:45.847254 30214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:45.876579 30214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:45.880496 30214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:45.880755 30214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:45.900601 30214 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:40379
I20260809 06:45:45.900609 30335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:40379 every 8 connection(s)
I20260809 06:45:45.901966 30214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:45:45.905614 30336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:45.911330 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30214
I20260809 06:45:45.911554 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:45:45.912199 28487 external_mini_cluster.cc:1428] Setting key 07a566f21072f3fe2f43f43106bb32c2
I20260809 06:45:45.915890 30336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: Bootstrap starting.
I20260809 06:45:45.919353 30336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:45.920370 30336 log.cc:824] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:45.923215 30336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: No bootstrap required, opened a new log
Aug 09 06:45:45 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:45.931255 30336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40379 } }
I20260809 06:45:45.931663 30336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:45.931764 30336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88cb5b530ecf4abbaaaa53dc17c0177f, State: Initialized, Role: FOLLOWER
I20260809 06:45:45.932372 30336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40379 } }
I20260809 06:45:45.932602 30336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:45:45.932741 30336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:45:45.932940 30336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:45.935086 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:45.914101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.27.209.254:40379)
0809 06:45:45.914885 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:45.914896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:45.914937 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:45:45.916573 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:45.916594 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:45.916953 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:45.917275 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:45.918958 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:45.920350 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:45.921816 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:45.922458 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:45.922585 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:45.927185 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:45.927194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:45.927210 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:45.927250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:45.930278 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:45.930951 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:45.930958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:45.930965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:45.931131 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:45.931488 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:45.931491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:45.931493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:45.932178 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:45.932448 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:45.933028 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 06:45:45.933687 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":334,"threads_started":1}
I20260809 06:45:45.936050 30336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40379 } }
I20260809 06:45:45.936625 30336 leader_election.cc:302] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88cb5b530ecf4abbaaaa53dc17c0177f; no voters:
I20260809 06:45:45.937255 30336 leader_election.cc:288] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:45:45.937378 30343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:45.938475 30343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [term 1 LEADER]: Becoming Leader. State: Replica: 88cb5b530ecf4abbaaaa53dc17c0177f, State: Running, Role: LEADER
I20260809 06:45:45.938948 30343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40379 } }
I20260809 06:45:45.939311 30336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:45:45.941454 30345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40379 } } }
I20260809 06:45:45.941833 30345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:45.942539 30346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88cb5b530ecf4abbaaaa53dc17c0177f. Latest consensus state: current_term: 1 leader_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88cb5b530ecf4abbaaaa53dc17c0177f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40379 } } }
I20260809 06:45:45.942734 30353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:45:45.942799 30346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:45.947881 30353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:45:45.956347 30353 catalog_manager.cc:1347] Generated new cluster ID: 3e96065d586449a5b55bf4117d317f77
I20260809 06:45:45.956457 30353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:45:45.974418 30353 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:45:45.976290 30353 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:45:45.995146 30353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: Generated new TSK 0
I20260809 06:45:45.996201 30353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:45:46.043565 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40379
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:46.375310 30378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:46.375651 30378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:46.375769 30378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:46.384298 30378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:46.384465 30378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:46.384590 30378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:45:46.384692 30378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:45:46.398947 30378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:40379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:46.400759 30378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:46.402684 30378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:46.417201 30388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:46.417244 30387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:46.417274 30390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:46.418224 30378 server_base.cc:1034] running on GCE node
I20260809 06:45:46.418787 30378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:46.419775 30378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:46.420976 30378 hybrid_clock.cc:648] HybridClock initialized: now 1786257946420906 us; error 48 us; skew 500 ppm
I20260809 06:45:46.421257 30378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:46 dist-test-slave-jpj1 krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257946, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:46.424292 30378 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:45:46.425616 30378 webserver.cc:460] Webserver started at http://127.27.209.193:33653/ using document root <none> and password file <none>
I20260809 06:45:46.426290 30378 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:46.426391 30378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:46.426697 30378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:46.429477 30378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "170e918a49da47ccafd36e8731f7e7ce"
format_stamp: "Formatted at 2026-08-09 06:45:46 on dist-test-slave-jpj1"
server_key: "6bffbeb9973f82292a423c2d86aa3d58"
server_key_iv: "b1a5a059d6da1c77285e7702bb513b3a"
server_key_version: "encryptionkey@0"
I20260809 06:45:46.430243 30378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "170e918a49da47ccafd36e8731f7e7ce"
format_stamp: "Formatted at 2026-08-09 06:45:46 on dist-test-slave-jpj1"
server_key: "6bffbeb9973f82292a423c2d86aa3d58"
server_key_iv: "b1a5a059d6da1c77285e7702bb513b3a"
server_key_version: "encryptionkey@0"
I20260809 06:45:46.435177 30378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:45:46.438390 30401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:46.439673 30378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:45:46.439822 30378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "170e918a49da47ccafd36e8731f7e7ce"
format_stamp: "Formatted at 2026-08-09 06:45:46 on dist-test-slave-jpj1"
server_key: "6bffbeb9973f82292a423c2d86aa3d58"
server_key_iv: "b1a5a059d6da1c77285e7702bb513b3a"
server_key_version: "encryptionkey@0"
I20260809 06:45:46.440002 30378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:46.477615 30378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:46.481766 30378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:46.482010 30378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:46.484660 30378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:46.484745 30378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:46.484913 30378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:46.485005 30378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:45:46 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257946, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:46.507424 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:46.485074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:59867 (local address 127.27.209.254:40379)
0809 06:45:46.485920 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:46.485927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:46.486794 (+ 867us) server_negotiation.cc:407] Connection header received
0809 06:45:46.489025 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:46.489031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:46.489121 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:46.489919 (+ 798us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:46.491741 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.492673 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.495449 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.495978 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.496113 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:46.500667 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:46.500674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:46.500680 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:46.500722 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:46.503919 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:46.504888 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:46.504893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:46.504897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:46.505058 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:46.505601 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:46.505606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:46.505610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:46.505998 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:46.506146 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:46.506787 (+ 641us) server_negotiation.cc:299] Negotiation successful
0809 06:45:46.507022 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":607,"threads_started":1}
I20260809 06:45:46.508121 30416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:46.485742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.27.209.193:59867)
0809 06:45:46.486609 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:46.486668 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:46.488706 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:46.489578 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:46.489609 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:46.490538 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:46.491481 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:46.491510 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.492829 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.492834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:46.495292 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:46.495312 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.496246 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.496254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:46.496558 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:46.497342 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:46.497384 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:46.500422 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:46.504088 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:46.504097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:46.504112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:46.504706 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:46.505158 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:46.505163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:46.505167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:46.505462 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:46.506147 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:46.506155 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:46.506454 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 06:45:46.506663 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 06:45:46.507005 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":350,"threads_started":1}
I20260809 06:45:46.525559 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:46.513646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.27.209.193:40601)
0809 06:45:46.514196 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:46.514211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:46.514315 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:46.514590 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:46.514592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:46.514991 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:46.515258 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:46.515266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.516591 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.516597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:46.518196 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:46.518212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.518942 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.518949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:46.519101 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:46.519676 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:46.519691 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:46.520400 (+ 709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:46.522827 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:46.522832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:46.522837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:46.523209 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:46.523591 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:46.523596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:46.523600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:46.523726 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:46.524284 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:46.524289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:46.524551 (+ 262us) client_negotiation.cc:769] Sending connection context
0809 06:45:46.524721 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 06:45:46.525024 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":287,"threads_started":1}
I20260809 06:45:46.526304 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:46.513586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:40601 (local address 127.27.209.254:40379)
0809 06:45:46.513735 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:46.513739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:46.514293 (+ 554us) server_negotiation.cc:407] Connection header received
0809 06:45:46.514401 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:46.514407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:46.514486 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:46.514584 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:46.515371 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.516450 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.518332 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.518764 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.518876 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:46.520570 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:46.520574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:46.520578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:46.520605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:46.522711 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:46.523316 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:46.523322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:46.523325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:46.523466 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:46.523844 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:46.523849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:46.523853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:46.524181 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:46.524318 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:46.525881 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0809 06:45:46.526076 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 06:45:46.536013 30378 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:32973
I20260809 06:45:46.536051 30599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:32973 every 8 connection(s)
I20260809 06:45:46.537153 30378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:45:46.539057 30600 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40379
I20260809 06:45:46.539263 30600 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:46.539937 30600 heartbeater.cc:511] Master 127.27.209.254:40379 requested a full tablet report, sending...
I20260809 06:45:46.541788 30264 ts_manager.cc:194] Registered new tserver with Master: 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193:32973)
I20260809 06:45:46.543278 30264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:59867
I20260809 06:45:46.544039 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30378
I20260809 06:45:46.544309 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:45:46.545001 28487 external_mini_cluster.cc:1428] Setting key 41d59493bd15a80300681607ac801772
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:45:46.584352 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40379
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:46.899744 30604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:46.900053 30604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:46.900195 30604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:46.907213 30604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:46.907343 30604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:46.907467 30604 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:45:46.907598 30604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:45:46.917361 30604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:40379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:46.918615 30604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:46.919951 30604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:46.932340 30616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:46.932296 30618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:46.932412 30615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:46.932999 30604 server_base.cc:1034] running on GCE node
I20260809 06:45:46.933625 30604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:46.934500 30604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:46.935693 30604 hybrid_clock.cc:648] HybridClock initialized: now 1786257946935632 us; error 41 us; skew 500 ppm
I20260809 06:45:46.935981 30604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:46 dist-test-slave-jpj1 krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257946, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:46.939018 30604 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:45:46.940440 30604 webserver.cc:460] Webserver started at http://127.27.209.194:43777/ using document root <none> and password file <none>
I20260809 06:45:46.941107 30604 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:46.941207 30604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:46.941560 30604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:46.944554 30604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "881b8526ec724c1aba58f37495de4f5e"
format_stamp: "Formatted at 2026-08-09 06:45:46 on dist-test-slave-jpj1"
server_key: "ee8a609bde06d278aed10d8ed604a22d"
server_key_iv: "838d7306265fe84444f7c166e589a59c"
server_key_version: "encryptionkey@0"
I20260809 06:45:46.945314 30604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "881b8526ec724c1aba58f37495de4f5e"
format_stamp: "Formatted at 2026-08-09 06:45:46 on dist-test-slave-jpj1"
server_key: "ee8a609bde06d278aed10d8ed604a22d"
server_key_iv: "838d7306265fe84444f7c166e589a59c"
server_key_version: "encryptionkey@0"
I20260809 06:45:46.950377 30604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:45:46.953788 30628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:46.955094 30604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:45:46.955247 30604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "881b8526ec724c1aba58f37495de4f5e"
format_stamp: "Formatted at 2026-08-09 06:45:46 on dist-test-slave-jpj1"
server_key: "ee8a609bde06d278aed10d8ed604a22d"
server_key_iv: "838d7306265fe84444f7c166e589a59c"
server_key_version: "encryptionkey@0"
I20260809 06:45:46.955483 30604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:46.976831 30604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:46.980803 30604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:46.981047 30604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:46.983762 30604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:46.983862 30604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:46.984002 30604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:46.984071 30604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:45:46 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257946, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:47.005995 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:46.984154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:45151 (local address 127.27.209.254:40379)
0809 06:45:46.984326 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:46.984333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:46.985880 (+ 1547us) server_negotiation.cc:407] Connection header received
0809 06:45:46.987912 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:46.987919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:46.987995 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:46.988088 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:46.990165 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.991115 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.993635 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.994072 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.994206 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:46.999108 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:46.999113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:46.999116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:46.999158 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:47.002162 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.003118 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.003124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.003127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.003289 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.003878 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.003884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.003888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.004278 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:47.004457 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.005403 (+ 946us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.005640 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 06:45:47.007066 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:46.984862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.27.209.194:45151)
0809 06:45:46.985703 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:46.985753 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:46.987601 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:46.988368 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:46.988380 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:46.989162 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:46.989974 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:46.989998 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.991279 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.991285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:46.993497 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:46.993510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:46.994386 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:46.994395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:46.994702 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:46.995746 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:46.995791 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:46.998905 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:47.002308 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.002317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.002338 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.002967 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.003436 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.003441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.003445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.003745 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.004441 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:47.004452 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:47.004910 (+ 458us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.005269 (+ 359us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.005827 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":366,"threads_started":1}
I20260809 06:45:47.025365 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.013330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:59967 (local address 127.27.209.254:40379)
0809 06:45:47.013514 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.013520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.013585 (+ 65us) server_negotiation.cc:407] Connection header received
0809 06:45:47.013715 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.013721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.013806 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.013910 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:47.014750 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.015936 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.017343 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.017998 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.018117 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.020212 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:47.020218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:47.020222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:47.020259 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:47.022716 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.023321 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.023326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.023330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.023460 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.023832 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.023837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.023840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.024280 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:47.024412 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.024821 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.025019 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:45:47.025559 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.012914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.27.209.194:59967)
0809 06:45:47.013496 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.013516 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.013622 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.013923 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.013926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.014358 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:47.014631 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.014641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.016181 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.016186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.017221 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.017233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.018174 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.018179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.018297 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.018888 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:47.018905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:47.020025 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:47.022838 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.022844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.022848 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.023213 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.023590 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.023595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.023599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.023724 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.024395 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:47.024399 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:47.024583 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.024718 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.025088 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":289,"threads_started":1}
I20260809 06:45:47.035845 30604 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:35733
I20260809 06:45:47.035864 30829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:35733 every 8 connection(s)
I20260809 06:45:47.037024 30604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:45:47.038950 30831 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40379
I20260809 06:45:47.039176 30831 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:47.039876 30831 heartbeater.cc:511] Master 127.27.209.254:40379 requested a full tablet report, sending...
I20260809 06:45:47.040313 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30604
I20260809 06:45:47.040500 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:45:47.041029 28487 external_mini_cluster.cc:1428] Setting key c4a04ab1f42cf85284fb27a4fc2e8807
I20260809 06:45:47.041333 30264 ts_manager.cc:194] Registered new tserver with Master: 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733)
I20260809 06:45:47.042245 30264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:45151
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:45:47.083406 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40379
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:44889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:47.405555 30837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:47.405870 30837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:47.406029 30837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:47.414497 30837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:47.414626 30837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:47.414763 30837 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:45:47.414882 30837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:45:47.425132 30837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:40379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:47.426471 30837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:47.427909 30837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:47.440946 30847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:47.440963 30848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:47.441054 30854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:47.442092 30837 server_base.cc:1034] running on GCE node
I20260809 06:45:47.442744 30837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:47.443801 30837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:47.445060 30837 hybrid_clock.cc:648] HybridClock initialized: now 1786257947444983 us; error 49 us; skew 500 ppm
I20260809 06:45:47.445351 30837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:47 dist-test-slave-jpj1 krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:47.448414 30837 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:45:47.449920 30837 webserver.cc:460] Webserver started at http://127.27.209.195:46425/ using document root <none> and password file <none>
I20260809 06:45:47.450621 30837 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:47.450731 30837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:47.451048 30837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:47.454154 30837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "b46bf234a3a5477fbe20436d0e1c6f0f"
format_stamp: "Formatted at 2026-08-09 06:45:47 on dist-test-slave-jpj1"
server_key: "872212c5bf793345032e62653ca06dca"
server_key_iv: "0a59ae80b2808a2217553413d0b780bd"
server_key_version: "encryptionkey@0"
I20260809 06:45:47.454955 30837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "b46bf234a3a5477fbe20436d0e1c6f0f"
format_stamp: "Formatted at 2026-08-09 06:45:47 on dist-test-slave-jpj1"
server_key: "872212c5bf793345032e62653ca06dca"
server_key_iv: "0a59ae80b2808a2217553413d0b780bd"
server_key_version: "encryptionkey@0"
I20260809 06:45:47.460374 30837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:45:47.464015 30863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:47.465400 30837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 06:45:47.465566 30837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "b46bf234a3a5477fbe20436d0e1c6f0f"
format_stamp: "Formatted at 2026-08-09 06:45:47 on dist-test-slave-jpj1"
server_key: "872212c5bf793345032e62653ca06dca"
server_key_iv: "0a59ae80b2808a2217553413d0b780bd"
server_key_version: "encryptionkey@0"
I20260809 06:45:47.465775 30837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:47.485823 30837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:47.490013 30837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:47.490283 30837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:47.493203 30837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:47.493324 30837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:47.493471 30837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:47.493551 30837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:45:47 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:47.516494 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.493699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:58559 (local address 127.27.209.254:40379)
0809 06:45:47.493903 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.493910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.495359 (+ 1449us) server_negotiation.cc:407] Connection header received
0809 06:45:47.497399 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.497404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.497489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.497577 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:47.499891 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.500892 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.503317 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.503913 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.504030 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.509877 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:47.509884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:47.509887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:47.509956 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:47.512542 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.513549 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.513553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.513556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.513706 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.514241 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.514245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.514248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.514620 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:47.514764 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.515796 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.516076 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:45:47.517282 30875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.494377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.27.209.195:58559)
0809 06:45:47.495187 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.495229 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.497117 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.497844 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.497853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.498601 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:47.499640 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.499670 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.501082 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.501089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.503181 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.503194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.504163 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.504171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.504459 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.506311 (+ 1852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:47.506352 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:47.509547 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:47.512691 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.512702 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.512732 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.513393 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.513816 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.513821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.513824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.514094 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.514757 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:47.514765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:47.515258 (+ 493us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.515641 (+ 383us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.516070 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":340,"threads_started":1}
I20260809 06:45:47.538487 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.522896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:57895 (local address 127.27.209.254:40379)
0809 06:45:47.523087 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.523092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.524008 (+ 916us) server_negotiation.cc:407] Connection header received
0809 06:45:47.524227 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.524232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.524327 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.524427 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:47.525760 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.526545 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.528014 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.528542 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.528670 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.531240 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:47.531248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:47.531254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:47.531297 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:47.534915 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.536023 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.536029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.536032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.536202 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.536704 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.536710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.536714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.537088 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:47.537241 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.537887 (+ 646us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.538137 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:45:47.538888 30940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.522895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.27.209.195:57895)
0809 06:45:47.523891 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.523912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.524081 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.524463 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.524468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.525117 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:47.525617 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.525630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.526690 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.526695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.527878 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.527897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.528753 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.528761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.528960 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.529695 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:47.529716 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:47.531012 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:47.535080 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.535090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.535095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.535858 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.536360 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:47.536365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:47.536370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:47.536549 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:47.537327 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:47.537333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:47.537587 (+ 254us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.537731 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.538103 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":642,"threads_started":1}
I20260809 06:45:47.546371 30600 heartbeater.cc:503] Master 127.27.209.254:40379 was elected leader, sending a full tablet report...
I20260809 06:45:47.548310 30837 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:40517
I20260809 06:45:47.548346 31046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:40517 every 8 connection(s)
I20260809 06:45:47.549435 30837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:45:47.550699 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30837
I20260809 06:45:47.550925 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:45:47.551371 31048 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40379
I20260809 06:45:47.551420 28487 external_mini_cluster.cc:1428] Setting key ad0838ef9553196f2904484f168a47e0
I20260809 06:45:47.551614 31048 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:47.552333 31048 heartbeater.cc:511] Master 127.27.209.254:40379 requested a full tablet report, sending...
I20260809 06:45:47.553612 30264 ts_manager.cc:194] Registered new tserver with Master: b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:47.554397 30264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:58559
I20260809 06:45:47.566241 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:45:47 dist-test-slave-jpj1 krb5kdc[30197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:45:47 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:47.612506 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.598842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.27.209.254:40379)
0809 06:45:47.599078 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.599084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.599243 (+ 159us) server_negotiation.cc:407] Connection header received
0809 06:45:47.599452 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.599456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.599523 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.599640 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:47.600957 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.601781 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.603294 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.603772 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.603871 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.607439 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:47.607443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:47.607447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:47.607478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:47.609993 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.610584 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.610589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.610592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.610719 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:47.611108 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:47.611114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:47.611117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:47.611384 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:47.611634 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.611936 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.612155 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 06:45:47.616485 30264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:45:47.623473 30264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:47.659101 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.639085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.27.209.193:32973)
0809 06:45:47.646827 (+ 7742us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.646837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.649488 (+ 2651us) server_negotiation.cc:407] Connection header received
0809 06:45:47.649868 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.649878 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.650268 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.651394 (+ 1126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.651635 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.653171 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.656050 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.657431 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.657605 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.657791 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.658523 (+ 732us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.658698 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7574,"thread_start_us":7431,"threads_started":1}
I20260809 06:45:47.659512 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.638183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:32973 (local address 127.0.0.1:42084)
0809 06:45:47.649300 (+ 11117us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.649340 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.649654 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.650577 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.650587 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.650626 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.651072 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.651085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.653372 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.653378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.655865 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.655879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.657780 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.657790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.658176 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.658259 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.658478 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.658887 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10851,"mutex_wait_us":1,"thread_start_us":580,"threads_started":1}
I20260809 06:45:47.661634 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.639860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:40517 (local address 127.0.0.1:52158)
0809 06:45:47.650875 (+ 11015us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.650906 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.651141 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.653946 (+ 2805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.653950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.653974 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.654733 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.654744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.656715 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.656719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.658997 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.659011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.660863 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.660869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.661043 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.661075 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.661151 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.661274 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10758,"mutex_wait_us":3,"thread_start_us":9980,"threads_started":1}
I20260809 06:45:47.662658 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.640155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.27.209.195:40517)
0809 06:45:47.653397 (+ 13242us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.653404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.653431 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:45:47.653507 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.653511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.653723 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.654094 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.654925 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.656575 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.659206 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.660628 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.662013 (+ 1385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.662167 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.662234 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.662386 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13070,"thread_start_us":12879,"threads_started":1}
I20260809 06:45:47.663461 30483 tablet_service.cc:1467] Processing CreateTablet for tablet 30d9db7b990b450d8fb354384ce43959 (DEFAULT_TABLE table=test-table [id=6a7e19621ee14c139a705fc66478cc21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:47.664409 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.637541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:35733 (local address 127.0.0.1:57136)
0809 06:45:47.638960 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.639007 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.639192 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.653280 (+ 14088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.653284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.653317 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.654729 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.654760 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.656666 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.656671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.658765 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.658780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.663645 (+ 4865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.663655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.663824 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.663855 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.663942 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.664064 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":47,"thread_start_us":14213,"threads_started":1}
I20260809 06:45:47.665045 30483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d9db7b990b450d8fb354384ce43959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:47.665077 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.637765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.27.209.194:35733)
0809 06:45:47.652097 (+ 14332us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.652120 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.652181 (+ 61us) server_negotiation.cc:407] Connection header received
0809 06:45:47.652372 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.652385 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.652859 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.654346 (+ 1487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.655008 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.656494 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.658944 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.660464 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.660616 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.660774 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.664632 (+ 3858us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.664789 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14030,"thread_start_us":13721,"threads_started":1}
I20260809 06:45:47.666725 30941 tablet_service.cc:1467] Processing CreateTablet for tablet 30d9db7b990b450d8fb354384ce43959 (DEFAULT_TABLE table=test-table [id=6a7e19621ee14c139a705fc66478cc21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:47.668334 30941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d9db7b990b450d8fb354384ce43959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:47.672124 30711 tablet_service.cc:1467] Processing CreateTablet for tablet 30d9db7b990b450d8fb354384ce43959 (DEFAULT_TABLE table=test-table [id=6a7e19621ee14c139a705fc66478cc21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:47.673504 30711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30d9db7b990b450d8fb354384ce43959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:47.678423 31073 tablet_bootstrap.cc:492] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Bootstrap starting.
I20260809 06:45:47.682469 31073 tablet_bootstrap.cc:654] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:47.683957 31073 log.cc:824] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:47.684269 31074 tablet_bootstrap.cc:492] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Bootstrap starting.
I20260809 06:45:47.687212 31073 tablet_bootstrap.cc:492] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: No bootstrap required, opened a new log
I20260809 06:45:47.687490 31073 ts_tablet_manager.cc:1397] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260809 06:45:47.687908 31074 tablet_bootstrap.cc:654] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:47.688108 31077 tablet_bootstrap.cc:492] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Bootstrap starting.
I20260809 06:45:47.688920 31074 log.cc:824] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:47.691438 31077 tablet_bootstrap.cc:654] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:47.691689 31074 tablet_bootstrap.cc:492] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: No bootstrap required, opened a new log
I20260809 06:45:47.691919 31074 ts_tablet_manager.cc:1397] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:45:47.692675 31077 log.cc:824] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:47.695514 31077 tablet_bootstrap.cc:492] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: No bootstrap required, opened a new log
I20260809 06:45:47.695824 31077 ts_tablet_manager.cc:1397] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260809 06:45:47.697568 31073 raft_consensus.cc:348] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.697992 31073 raft_consensus.cc:374] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:47.698101 31073 raft_consensus.cc:729] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e918a49da47ccafd36e8731f7e7ce, State: Initialized, Role: FOLLOWER
I20260809 06:45:47.698699 31073 consensus_queue.cc:260] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.699316 31074 raft_consensus.cc:348] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.699726 31074 raft_consensus.cc:374] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:47.699823 31074 raft_consensus.cc:729] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46bf234a3a5477fbe20436d0e1c6f0f, State: Initialized, Role: FOLLOWER
I20260809 06:45:47.700368 31074 consensus_queue.cc:260] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.700577 31073 ts_tablet_manager.cc:1428] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 06:45:47.702752 31048 heartbeater.cc:503] Master 127.27.209.254:40379 was elected leader, sending a full tablet report...
I20260809 06:45:47.703397 31074 ts_tablet_manager.cc:1428] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 06:45:47.705421 31077 raft_consensus.cc:348] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.705801 31077 raft_consensus.cc:374] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:47.705904 31077 raft_consensus.cc:729] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881b8526ec724c1aba58f37495de4f5e, State: Initialized, Role: FOLLOWER
I20260809 06:45:47.706393 31077 consensus_queue.cc:260] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.707334 30831 heartbeater.cc:503] Master 127.27.209.254:40379 was elected leader, sending a full tablet report...
I20260809 06:45:47.707970 31077 ts_tablet_manager.cc:1428] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260809 06:45:47.776168 31081 raft_consensus.cc:482] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:47.776465 31081 raft_consensus.cc:504] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.778126 31081 leader_election.cc:288] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733), b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:47.786573 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.778530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:47957 (local address 127.27.209.194:35733)
0809 06:45:47.778867 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.778873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.779278 (+ 405us) server_negotiation.cc:407] Connection header received
0809 06:45:47.779470 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.779474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.779581 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.779679 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.780289 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.781389 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.784324 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.785509 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.785669 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.785742 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.786188 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.786315 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260809 06:45:47.786564 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.778483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:52005 (local address 127.27.209.195:40517)
0809 06:45:47.778678 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.778685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.779483 (+ 798us) server_negotiation.cc:407] Connection header received
0809 06:45:47.779644 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.779649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.779724 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.779809 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.780428 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.781454 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.784337 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.785509 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.785669 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.785742 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.786192 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.786325 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:45:47.787037 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.778361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:35733 (local address 127.27.209.193:47957)
0809 06:45:47.779144 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.779168 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.779335 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.779691 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.779694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.779714 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.780146 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.780159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.781514 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.781519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.784171 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.784187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.785795 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.785804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.786002 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.786055 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.786154 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.786313 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":152,"thread_start_us":440,"threads_started":1}
I20260809 06:45:47.787077 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.778505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:40517 (local address 127.27.209.193:52005)
0809 06:45:47.779385 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.779399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.779500 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.779845 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.779849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.779873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.780274 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.780285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.781564 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.781571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.784180 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.784189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.785827 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.785833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.786009 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.786057 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.786152 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.786313 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":209,"thread_start_us":607,"threads_started":1}
I20260809 06:45:47.787945 30976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d9db7b990b450d8fb354384ce43959" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" is_pre_election: true
I20260809 06:45:47.788007 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d9db7b990b450d8fb354384ce43959" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881b8526ec724c1aba58f37495de4f5e" is_pre_election: true
I20260809 06:45:47.788420 30976 raft_consensus.cc:2454] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 0.
I20260809 06:45:47.788439 30753 raft_consensus.cc:2454] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 0.
W20260809 06:45:47.788849 30833 tablet.cc:2369] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:47.789099 30403 leader_election.cc:302] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e918a49da47ccafd36e8731f7e7ce, 881b8526ec724c1aba58f37495de4f5e; no voters:
W20260809 06:45:47.789261 30601 tablet.cc:2369] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:47.789809 31081 raft_consensus.cc:2781] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:47.789932 31081 raft_consensus.cc:482] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:47.790028 31081 raft_consensus.cc:3037] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:47.792233 31081 raft_consensus.cc:504] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.792847 31081 leader_election.cc:288] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 election: Requested vote from peers 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733), b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:47.793192 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d9db7b990b450d8fb354384ce43959" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881b8526ec724c1aba58f37495de4f5e"
I20260809 06:45:47.793335 30976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30d9db7b990b450d8fb354384ce43959" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f"
I20260809 06:45:47.793396 30753 raft_consensus.cc:3037] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:47.793531 30976 raft_consensus.cc:3037] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:47.796573 30976 raft_consensus.cc:2454] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 1.
I20260809 06:45:47.796568 30753 raft_consensus.cc:2454] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 1.
I20260809 06:45:47.796967 30403 leader_election.cc:302] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 170e918a49da47ccafd36e8731f7e7ce, 881b8526ec724c1aba58f37495de4f5e, b46bf234a3a5477fbe20436d0e1c6f0f; no voters:
I20260809 06:45:47.797245 31081 raft_consensus.cc:2781] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:47.797663 31081 raft_consensus.cc:686] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 1 LEADER]: Becoming Leader. State: Replica: 170e918a49da47ccafd36e8731f7e7ce, State: Running, Role: LEADER
I20260809 06:45:47.798146 31081 consensus_queue.cc:237] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
W20260809 06:45:47.802145 31049 tablet.cc:2369] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:47.803035 30263 catalog_manager.cc:5495] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193). New cstate: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } health_report { overall_health: UNKNOWN } } }
I20260809 06:45:47.857749 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.852860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.27.209.193:32973)
0809 06:45:47.853083 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.853088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.853109 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:45:47.853377 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.853380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.853457 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.853571 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:47.853852 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.854534 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.855839 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.856246 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.856337 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.856733 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:47.857211 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:47.857310 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.857396 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.857463 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:45:47.865716 30976 raft_consensus.cc:1261] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Refusing update from remote peer 170e918a49da47ccafd36e8731f7e7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:47.865712 30753 raft_consensus.cc:1261] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Refusing update from remote peer 170e918a49da47ccafd36e8731f7e7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:47.866628 31087 consensus_queue.cc:1035] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:47.867002 31081 consensus_queue.cc:1035] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:47.883385 31090 mvcc.cc:205] Tried to move back new op lower bound from 7316512554448826368 to 7316512554185011200. Current Snapshot: MvccSnapshot[applied={T|T < 7316512554448826368}]
I20260809 06:45:47.902350 30263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:47.902709 30263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:47.907007 30263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: Sending DeleteTablet for 3 replicas of tablet 30d9db7b990b450d8fb354384ce43959
I20260809 06:45:47.908221 30483 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d9db7b990b450d8fb354384ce43959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:47 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42084
I20260809 06:45:47.908242 30711 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d9db7b990b450d8fb354384ce43959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:47 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:57136
I20260809 06:45:47.909901 30941 tablet_service.cc:1514] Processing DeleteTablet for tablet 30d9db7b990b450d8fb354384ce43959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:47 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:52158
I20260809 06:45:47.918741 30264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:45:47.919459 30264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:47.921007 31108 tablet_replica.cc:330] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: stopping tablet replica
I20260809 06:45:47.921643 31108 raft_consensus.cc:2229] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:47.922251 31108 raft_consensus.cc:2258] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:47.924548 31108 ts_tablet_manager.cc:1905] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:47.928027 30935 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0353d4511446b49465c1731a4fda59 (DEFAULT_TABLE table=test-table [id=a84e2fe7637a4c74a30a9e88588239f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:47.928682 30935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0353d4511446b49465c1731a4fda59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:47.931954 31108 ts_tablet_manager.cc:1918] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:45:47.932205 31108 log.cc:1196] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/30d9db7b990b450d8fb354384ce43959
I20260809 06:45:47.932838 31108 ts_tablet_manager.cc:1939] T 30d9db7b990b450d8fb354384ce43959 P 881b8526ec724c1aba58f37495de4f5e: Deleting consensus metadata
I20260809 06:45:47.932785 30481 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0353d4511446b49465c1731a4fda59 (DEFAULT_TABLE table=test-table [id=a84e2fe7637a4c74a30a9e88588239f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:47.933430 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0353d4511446b49465c1731a4fda59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:47.935361 31109 tablet_replica.cc:330] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: stopping tablet replica
I20260809 06:45:47.935902 31109 raft_consensus.cc:2229] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:47.936472 31109 raft_consensus.cc:2258] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:47.937073 31106 tablet_replica.cc:330] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: stopping tablet replica
I20260809 06:45:47.939616 30237 catalog_manager.cc:4841] TS 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733): tablet 30d9db7b990b450d8fb354384ce43959 (table test-table [id=6a7e19621ee14c139a705fc66478cc21]) successfully deleted
I20260809 06:45:47.939992 30711 tablet_service.cc:1467] Processing CreateTablet for tablet 7d0353d4511446b49465c1731a4fda59 (DEFAULT_TABLE table=test-table [id=a84e2fe7637a4c74a30a9e88588239f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:47.940415 30711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d0353d4511446b49465c1731a4fda59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:47.941706 31074 tablet_bootstrap.cc:492] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Bootstrap starting.
I20260809 06:45:47.940336 31073 tablet_bootstrap.cc:492] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Bootstrap starting.
I20260809 06:45:47.942117 31109 ts_tablet_manager.cc:1905] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:47.943337 31074 tablet_bootstrap.cc:654] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:47.943482 31073 tablet_bootstrap.cc:654] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:47.947001 31077 tablet_bootstrap.cc:492] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Bootstrap starting.
I20260809 06:45:47.948589 31077 tablet_bootstrap.cc:654] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:47.949216 31109 ts_tablet_manager.cc:1918] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:45:47.949375 31109 log.cc:1196] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/30d9db7b990b450d8fb354384ce43959
I20260809 06:45:47.949941 31109 ts_tablet_manager.cc:1939] T 30d9db7b990b450d8fb354384ce43959 P b46bf234a3a5477fbe20436d0e1c6f0f: Deleting consensus metadata
I20260809 06:45:47.951815 30245 catalog_manager.cc:4841] TS b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517): tablet 30d9db7b990b450d8fb354384ce43959 (table test-table [id=6a7e19621ee14c139a705fc66478cc21]) successfully deleted
I20260809 06:45:47.952088 31106 raft_consensus.cc:2229] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:45:47.953177 31106 raft_consensus.cc:2258] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:47.953496 31077 tablet_bootstrap.cc:492] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: No bootstrap required, opened a new log
I20260809 06:45:47.953644 31077 ts_tablet_manager.cc:1397] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 06:45:47.954450 31074 tablet_bootstrap.cc:492] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: No bootstrap required, opened a new log
I20260809 06:45:47.954430 31077 raft_consensus.cc:348] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.954579 31074 ts_tablet_manager.cc:1397] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260809 06:45:47.954646 31077 raft_consensus.cc:374] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:47.954735 31077 raft_consensus.cc:729] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881b8526ec724c1aba58f37495de4f5e, State: Initialized, Role: FOLLOWER
I20260809 06:45:47.954986 31077 consensus_queue.cc:260] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.955410 31074 raft_consensus.cc:348] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.955654 31074 raft_consensus.cc:374] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:47.955758 31074 raft_consensus.cc:729] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46bf234a3a5477fbe20436d0e1c6f0f, State: Initialized, Role: FOLLOWER
I20260809 06:45:47.955788 31106 ts_tablet_manager.cc:1905] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:47.956064 31074 consensus_queue.cc:260] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.956727 31074 ts_tablet_manager.cc:1428] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:45:47.957377 31077 ts_tablet_manager.cc:1428] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 06:45:47.959097 31073 tablet_bootstrap.cc:492] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: No bootstrap required, opened a new log
I20260809 06:45:47.959251 31073 ts_tablet_manager.cc:1397] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260809 06:45:47.960059 31073 raft_consensus.cc:348] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.960294 31073 raft_consensus.cc:374] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:47.960386 31073 raft_consensus.cc:729] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e918a49da47ccafd36e8731f7e7ce, State: Initialized, Role: FOLLOWER
I20260809 06:45:47.960683 31073 consensus_queue.cc:260] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.961352 31073 ts_tablet_manager.cc:1428] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:45:47.966161 31106 ts_tablet_manager.cc:1918] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:45:47.966295 31106 log.cc:1196] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/30d9db7b990b450d8fb354384ce43959
I20260809 06:45:47.966755 31106 ts_tablet_manager.cc:1939] T 30d9db7b990b450d8fb354384ce43959 P 170e918a49da47ccafd36e8731f7e7ce: Deleting consensus metadata
I20260809 06:45:47.968117 30245 catalog_manager.cc:4841] TS 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193:32973): tablet 30d9db7b990b450d8fb354384ce43959 (table test-table [id=6a7e19621ee14c139a705fc66478cc21]) successfully deleted
I20260809 06:45:47.990087 31083 raft_consensus.cc:482] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:47.990288 31083 raft_consensus.cc:504] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:47.991822 31083 leader_election.cc:288] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193:32973), b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:47.999164 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.992115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:32973 (local address 127.27.209.194:45763)
0809 06:45:47.992620 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.992635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.992762 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.993335 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.993339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.993372 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.993737 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.993750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.995063 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.995067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.996894 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.996910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.998447 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.998453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.998596 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.998620 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:45:47.998700 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 06:45:47.998823 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260809 06:45:47.999182 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.992197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:45763 (local address 127.27.209.193:32973)
0809 06:45:47.992378 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.992385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.992720 (+ 335us) server_negotiation.cc:407] Connection header received
0809 06:45:47.992945 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.992951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.993046 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.993110 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.993940 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.994936 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.997097 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.998270 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.998418 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.998500 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:47.998740 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 06:45:47.998879 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 06:45:48.000226 30522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0353d4511446b49465c1731a4fda59" candidate_uuid: "881b8526ec724c1aba58f37495de4f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e918a49da47ccafd36e8731f7e7ce" is_pre_election: true
I20260809 06:45:48.000474 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.992791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:40517 (local address 127.27.209.194:38981)
0809 06:45:47.993608 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:47.993625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:47.993758 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:47.994091 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:47.994095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:47.994121 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:47.994492 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.994503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.995944 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.995949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.998030 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:47.998046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.999780 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.999785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:47.999932 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.999957 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:45:48.000040 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 06:45:48.000162 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":486,"threads_started":1}
I20260809 06:45:48.000468 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:47.992964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:38981 (local address 127.27.209.195:40517)
0809 06:45:47.993130 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:47.993135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:47.993728 (+ 593us) server_negotiation.cc:407] Connection header received
0809 06:45:47.993886 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:47.993891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:47.993967 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:47.994063 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:47.994673 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.995795 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.998194 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:47.999382 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:47.999529 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:47.999634 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.000085 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.000216 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:45:48.000695 30522 raft_consensus.cc:2454] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881b8526ec724c1aba58f37495de4f5e in term 0.
I20260809 06:45:48.001202 30976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0353d4511446b49465c1731a4fda59" candidate_uuid: "881b8526ec724c1aba58f37495de4f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" is_pre_election: true
I20260809 06:45:48.001415 30976 raft_consensus.cc:2454] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881b8526ec724c1aba58f37495de4f5e in term 0.
I20260809 06:45:48.001858 30635 leader_election.cc:302] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e918a49da47ccafd36e8731f7e7ce, 881b8526ec724c1aba58f37495de4f5e; no voters:
I20260809 06:45:48.002259 31083 raft_consensus.cc:2781] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:48.002381 31083 raft_consensus.cc:482] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:48.002488 31083 raft_consensus.cc:3037] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:48.004612 31083 raft_consensus.cc:504] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.005147 31083 leader_election.cc:288] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [CANDIDATE]: Term 1 election: Requested vote from peers 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193:32973), b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:48.005607 30522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0353d4511446b49465c1731a4fda59" candidate_uuid: "881b8526ec724c1aba58f37495de4f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e918a49da47ccafd36e8731f7e7ce"
I20260809 06:45:48.005679 30976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d0353d4511446b49465c1731a4fda59" candidate_uuid: "881b8526ec724c1aba58f37495de4f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f"
I20260809 06:45:48.005831 30522 raft_consensus.cc:3037] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:48.005861 30976 raft_consensus.cc:3037] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:48.007843 30976 raft_consensus.cc:2454] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881b8526ec724c1aba58f37495de4f5e in term 1.
I20260809 06:45:48.008200 30635 leader_election.cc:302] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881b8526ec724c1aba58f37495de4f5e, b46bf234a3a5477fbe20436d0e1c6f0f; no voters:
I20260809 06:45:48.008466 31083 raft_consensus.cc:2781] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:48.008932 31083 raft_consensus.cc:686] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 1 LEADER]: Becoming Leader. State: Replica: 881b8526ec724c1aba58f37495de4f5e, State: Running, Role: LEADER
I20260809 06:45:48.009435 30522 raft_consensus.cc:2454] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881b8526ec724c1aba58f37495de4f5e in term 1.
I20260809 06:45:48.009570 31083 consensus_queue.cc:237] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.013393 30263 catalog_manager.cc:5495] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 881b8526ec724c1aba58f37495de4f5e (127.27.209.194). New cstate: current_term: 1 leader_uuid: "881b8526ec724c1aba58f37495de4f5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257945, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:48.076581 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.062409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40379 (local address 127.0.0.1:45154)
0809 06:45:48.063061 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:48.063077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:48.063239 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:48.063583 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:48.063588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:48.064329 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:48.064625 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.064636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.065620 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.065625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.066644 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.066660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.067373 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.067380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.067561 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.068303 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:48.068320 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:48.070693 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:48.073670 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:48.073683 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:48.073697 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:48.074106 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:48.074471 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:48.074475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:48.074480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:48.074620 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:48.075260 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:48.075270 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:48.075562 (+ 292us) client_negotiation.cc:769] Sending connection context
0809 06:45:48.075796 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 06:45:48.076094 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":327,"threads_started":1}
I20260809 06:45:48.077219 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.062133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.27.209.254:40379)
0809 06:45:48.062319 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.062324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.063163 (+ 839us) server_negotiation.cc:407] Connection header received
0809 06:45:48.063354 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.063357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.063433 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.063516 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:48.064774 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.065496 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.066783 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.067203 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.067313 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.070860 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:48.070865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:48.070868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:48.070895 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:48.073535 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.074225 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.074230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.074233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.074354 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.074727 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.074731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.074733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.075083 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:48.075231 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.075980 (+ 749us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.076360 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:45:48.082823 30263 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:45154:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 06:45:48.084093 30263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:48.091050 30711 tablet_service.cc:1467] Processing CreateTablet for tablet b129a1797c0144e9ace9c61b40e5b861 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56605d62d14542be8bb75d7d16f157ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:45:48.091044 30481 tablet_service.cc:1467] Processing CreateTablet for tablet b129a1797c0144e9ace9c61b40e5b861 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56605d62d14542be8bb75d7d16f157ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:45:48.091605 30711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b129a1797c0144e9ace9c61b40e5b861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:48.091629 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b129a1797c0144e9ace9c61b40e5b861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:48.095824 30935 tablet_service.cc:1467] Processing CreateTablet for tablet b129a1797c0144e9ace9c61b40e5b861 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56605d62d14542be8bb75d7d16f157ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:45:48.096415 30935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b129a1797c0144e9ace9c61b40e5b861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:48.097505 31077 tablet_bootstrap.cc:492] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: Bootstrap starting.
I20260809 06:45:48.099045 31077 tablet_bootstrap.cc:654] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:48.101362 31077 tablet_bootstrap.cc:492] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: No bootstrap required, opened a new log
I20260809 06:45:48.101527 31077 ts_tablet_manager.cc:1397] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 06:45:48.102155 31077 raft_consensus.cc:348] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.102358 31077 raft_consensus.cc:374] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:48.102439 31077 raft_consensus.cc:729] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881b8526ec724c1aba58f37495de4f5e, State: Initialized, Role: FOLLOWER
I20260809 06:45:48.102675 31077 consensus_queue.cc:260] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.103060 31083 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.103348 31083 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.104043 31077 ts_tablet_manager.cc:1428] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 06:45:48.107420 31073 tablet_bootstrap.cc:492] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: Bootstrap starting.
I20260809 06:45:48.107668 31074 tablet_bootstrap.cc:492] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: Bootstrap starting.
I20260809 06:45:48.108913 31073 tablet_bootstrap.cc:654] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:48.109319 31074 tablet_bootstrap.cc:654] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:48.118201 31073 tablet_bootstrap.cc:492] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: No bootstrap required, opened a new log
I20260809 06:45:48.118395 31073 ts_tablet_manager.cc:1397] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260809 06:45:48.118737 31074 tablet_bootstrap.cc:492] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: No bootstrap required, opened a new log
I20260809 06:45:48.118909 31074 ts_tablet_manager.cc:1397] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260809 06:45:48.119138 31073 raft_consensus.cc:348] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.119405 31073 raft_consensus.cc:374] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:48.119500 31073 raft_consensus.cc:729] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e918a49da47ccafd36e8731f7e7ce, State: Initialized, Role: FOLLOWER
I20260809 06:45:48.119640 31074 raft_consensus.cc:348] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.119884 31074 raft_consensus.cc:374] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:48.119779 31073 consensus_queue.cc:260] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.119958 31074 raft_consensus.cc:729] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46bf234a3a5477fbe20436d0e1c6f0f, State: Initialized, Role: FOLLOWER
I20260809 06:45:48.120225 31074 consensus_queue.cc:260] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.120292 31081 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.120544 31073 ts_tablet_manager.cc:1428] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 06:45:48.120595 31081 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.120649 31082 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.120942 31082 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.121788 31074 ts_tablet_manager.cc:1428] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 06:45:48.186211 31081 raft_consensus.cc:482] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:48.186509 31081 raft_consensus.cc:504] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.187387 31081 leader_election.cc:288] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733), b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:48.187971 30976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b129a1797c0144e9ace9c61b40e5b861" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" is_pre_election: true
I20260809 06:45:48.188019 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b129a1797c0144e9ace9c61b40e5b861" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881b8526ec724c1aba58f37495de4f5e" is_pre_election: true
I20260809 06:45:48.188251 30976 raft_consensus.cc:2454] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 0.
I20260809 06:45:48.188313 30753 raft_consensus.cc:2454] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 0.
I20260809 06:45:48.188750 30406 leader_election.cc:302] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e918a49da47ccafd36e8731f7e7ce, b46bf234a3a5477fbe20436d0e1c6f0f; no voters:
I20260809 06:45:48.189109 31081 raft_consensus.cc:2781] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:48.189261 31081 raft_consensus.cc:482] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:48.189384 31081 raft_consensus.cc:3037] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:48.192778 31081 raft_consensus.cc:504] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.193480 31081 leader_election.cc:288] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 election: Requested vote from peers 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733), b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517)
I20260809 06:45:48.193962 30976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b129a1797c0144e9ace9c61b40e5b861" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f"
I20260809 06:45:48.193989 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b129a1797c0144e9ace9c61b40e5b861" candidate_uuid: "170e918a49da47ccafd36e8731f7e7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881b8526ec724c1aba58f37495de4f5e"
I20260809 06:45:48.194188 30976 raft_consensus.cc:3037] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:48.194216 30753 raft_consensus.cc:3037] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:48.196496 30976 raft_consensus.cc:2454] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 1.
I20260809 06:45:48.196902 30406 leader_election.cc:302] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e918a49da47ccafd36e8731f7e7ce, b46bf234a3a5477fbe20436d0e1c6f0f; no voters:
I20260809 06:45:48.197202 31081 raft_consensus.cc:2781] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:48.197352 31081 raft_consensus.cc:686] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [term 1 LEADER]: Becoming Leader. State: Replica: 170e918a49da47ccafd36e8731f7e7ce, State: Running, Role: LEADER
I20260809 06:45:48.197295 30753 raft_consensus.cc:2454] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e918a49da47ccafd36e8731f7e7ce in term 1.
I20260809 06:45:48.197685 31081 consensus_queue.cc:237] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } }
I20260809 06:45:48.199579 31087 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: TxnStatusTablet state changed. Reason: New leader 170e918a49da47ccafd36e8731f7e7ce. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.199956 31087 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:45:48.200678 31136 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 06:45:48.200906 31136 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 06:45:48.201455 30262 catalog_manager.cc:5495] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193). New cstate: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } health_report { overall_health: UNKNOWN } } }
I20260809 06:45:48.228374 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.222646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:32973 (local address 127.0.0.1:42096)
0809 06:45:48.223042 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:48.223058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:48.223191 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:48.223672 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:48.223676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:48.223686 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:48.223850 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.223857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.225310 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.225313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.226259 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.226270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.227277 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.227280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.227415 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.227469 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.227955 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.227968 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 06:45:48.228028 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 06:45:48.228140 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260809 06:45:48.228468 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.222716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.27.209.193:32973)
0809 06:45:48.222935 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.222942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.223141 (+ 199us) server_negotiation.cc:407] Connection header received
0809 06:45:48.223369 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.223380 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.223472 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.223657 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:48.223998 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.225198 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.226421 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.227118 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.227271 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.227725 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.227862 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.227958 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.228069 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.228157 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:45:48.231870 30753 raft_consensus.cc:1261] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Refusing update from remote peer 170e918a49da47ccafd36e8731f7e7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:48.231935 30976 raft_consensus.cc:1261] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Refusing update from remote peer 170e918a49da47ccafd36e8731f7e7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:48.232474 31087 consensus_queue.cc:1035] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:48.232767 31081 consensus_queue.cc:1035] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:48.242722 31082 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: TxnStatusTablet state changed. Reason: New leader 170e918a49da47ccafd36e8731f7e7ce. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.243003 31082 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.244413 31087 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.244680 31087 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:45:48.248449 31082 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.248682 31082 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P b46bf234a3a5477fbe20436d0e1c6f0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.249541 31083 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: TxnStatusTablet state changed. Reason: New leader 170e918a49da47ccafd36e8731f7e7ce. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.249863 31083 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.251374 31081 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.251689 31081 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 170e918a49da47ccafd36e8731f7e7ce: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:45:48.253997 31083 tablet_replica.cc:439] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170e918a49da47ccafd36e8731f7e7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 } } peers { permanent_uuid: "881b8526ec724c1aba58f37495de4f5e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35733 } } peers { permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 } } }
I20260809 06:45:48.254287 31083 tablet_replica.cc:442] T b129a1797c0144e9ace9c61b40e5b861 P 881b8526ec724c1aba58f37495de4f5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:48.260118 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.252934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.27.209.194:35733)
0809 06:45:48.253114 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.253120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.253138 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:45:48.253251 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.253257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.253364 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.253524 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:48.254174 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.255479 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.257705 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.258123 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.258225 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.258817 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.259440 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.259633 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.259730 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.259802 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":40576}
I20260809 06:45:48.277185 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.270805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:32973 (local address 127.27.209.194:59395)
0809 06:45:48.271459 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:48.271475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:48.271670 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:48.272042 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:48.272046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:48.272055 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:48.272325 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.272333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.273860 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.273864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.274962 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.274975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.276076 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.276081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.276213 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.276269 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.276713 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.276728 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 06:45:48.276781 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 06:45:48.276892 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":341,"threads_started":1}
I20260809 06:45:48.277249 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.270816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:59395 (local address 127.27.209.193:32973)
0809 06:45:48.270991 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.270998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.271584 (+ 586us) server_negotiation.cc:407] Connection header received
0809 06:45:48.271818 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.271824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.271920 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.272029 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:48.272501 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.273703 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.275130 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.275880 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.276013 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.276448 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.276624 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.276719 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.276838 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.276924 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:45:48.286504 30976 raft_consensus.cc:1261] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Refusing update from remote peer 881b8526ec724c1aba58f37495de4f5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:48.286839 30522 raft_consensus.cc:1261] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Refusing update from remote peer 881b8526ec724c1aba58f37495de4f5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:48.287657 31118 consensus_queue.cc:1035] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46bf234a3a5477fbe20436d0e1c6f0f" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:48.288108 31083 consensus_queue.cc:1035] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "170e918a49da47ccafd36e8731f7e7ce" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:48.293322 31094 mvcc.cc:205] Tried to move back new op lower bound from 7316512556175659008 to 7316512555052298240. Current Snapshot: MvccSnapshot[applied={T|T < 7316512556175659008}]
I20260809 06:45:48.324218 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.317052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:53527 (local address 127.27.209.194:35733)
0809 06:45:48.317254 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.317260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.317741 (+ 481us) server_negotiation.cc:407] Connection header received
0809 06:45:48.317972 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.317977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.318064 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.318147 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:48.318636 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.319681 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.322022 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.322694 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.322798 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.323471 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.323646 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.323711 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.323926 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.323995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:45:48.324460 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.316931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:35733 (local address 127.27.209.193:53527)
0809 06:45:48.317627 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:48.317652 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:48.317823 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:48.318168 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:48.318172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:48.318184 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:48.318474 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.318485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.319819 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.319824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.321841 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:48.321859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.322987 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.322994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:48.323164 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.323253 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:48.323798 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:48.323808 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 06:45:48.323884 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 06:45:48.324042 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":299,"threads_started":1}
I20260809 06:45:48.364380 30263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:48.364662 30263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:48.368114 30263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88cb5b530ecf4abbaaaa53dc17c0177f: Sending DeleteTablet for 3 replicas of tablet 7d0353d4511446b49465c1731a4fda59
I20260809 06:45:48.369040 30935 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0353d4511446b49465c1731a4fda59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:48 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:52158
I20260809 06:45:48.369344 31109 tablet_replica.cc:330] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: stopping tablet replica
I20260809 06:45:48.369311 30711 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0353d4511446b49465c1731a4fda59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:48 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:57136
I20260809 06:45:48.369535 31109 raft_consensus.cc:2229] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:48.369604 31108 tablet_replica.cc:330] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: stopping tablet replica
I20260809 06:45:48.369725 31109 raft_consensus.cc:2258] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:48.369810 31108 raft_consensus.cc:2229] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:45:48.370311 31108 raft_consensus.cc:2258] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:48.371937 30481 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d0353d4511446b49465c1731a4fda59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:48 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42084
I20260809 06:45:48.372496 31106 tablet_replica.cc:330] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: stopping tablet replica
I20260809 06:45:48.372709 31106 raft_consensus.cc:2229] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:48.372920 31106 raft_consensus.cc:2258] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:48.374351 31106 ts_tablet_manager.cc:1905] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:48.380380 31108 ts_tablet_manager.cc:1905] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:48.371498 31109 ts_tablet_manager.cc:1905] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:48.391831 31108 ts_tablet_manager.cc:1918] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:48.397734 31106 ts_tablet_manager.cc:1918] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:48.397908 31109 ts_tablet_manager.cc:1918] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:48.398128 31109 log.cc:1196] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/7d0353d4511446b49465c1731a4fda59
I20260809 06:45:48.398640 31109 ts_tablet_manager.cc:1939] T 7d0353d4511446b49465c1731a4fda59 P b46bf234a3a5477fbe20436d0e1c6f0f: Deleting consensus metadata
I20260809 06:45:48.399907 30245 catalog_manager.cc:4841] TS b46bf234a3a5477fbe20436d0e1c6f0f (127.27.209.195:40517): tablet 7d0353d4511446b49465c1731a4fda59 (table test-table [id=a84e2fe7637a4c74a30a9e88588239f9]) successfully deleted
I20260809 06:45:48.400321 31106 log.cc:1196] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/7d0353d4511446b49465c1731a4fda59
I20260809 06:45:48.400621 31108 log.cc:1196] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/7d0353d4511446b49465c1731a4fda59
I20260809 06:45:48.401109 31106 ts_tablet_manager.cc:1939] T 7d0353d4511446b49465c1731a4fda59 P 170e918a49da47ccafd36e8731f7e7ce: Deleting consensus metadata
I20260809 06:45:48.401662 31108 ts_tablet_manager.cc:1939] T 7d0353d4511446b49465c1731a4fda59 P 881b8526ec724c1aba58f37495de4f5e: Deleting consensus metadata
I20260809 06:45:48.402714 30245 catalog_manager.cc:4841] TS 170e918a49da47ccafd36e8731f7e7ce (127.27.209.193:32973): tablet 7d0353d4511446b49465c1731a4fda59 (table test-table [id=a84e2fe7637a4c74a30a9e88588239f9]) successfully deleted
I20260809 06:45:48.404107 30237 catalog_manager.cc:4841] TS 881b8526ec724c1aba58f37495de4f5e (127.27.209.194:35733): tablet 7d0353d4511446b49465c1731a4fda59 (table test-table [id=a84e2fe7637a4c74a30a9e88588239f9]) successfully deleted
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:45:48.424691 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.383747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.27.209.193:32973)
0809 06:45:48.383923 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.383929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.405101 (+ 21172us) server_negotiation.cc:407] Connection header received
0809 06:45:48.405382 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.405388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.405465 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.405529 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:48.407628 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.408742 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.410818 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.411519 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.411688 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.417031 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:48.417048 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:48.417070 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:48.417129 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:48.421351 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.422234 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.422244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.422248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.422414 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.422853 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.422859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.422863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.423375 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:48.423797 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.424046 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.424312 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 06:45:48.425921 30594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42112
I20260809 06:45:48.446113 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.429922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.27.209.254:40379)
0809 06:45:48.430152 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.430157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.430400 (+ 243us) server_negotiation.cc:407] Connection header received
0809 06:45:48.430581 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.430587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.430674 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.430773 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:48.432555 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.433761 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.435705 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.436384 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.436505 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.439392 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:48.439399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:48.439404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:48.439443 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:48.442911 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.443802 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.443808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.443812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.443992 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.444509 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.444515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.444518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.444931 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:48.445215 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.445518 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.445746 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260809 06:45:48.446997 30263 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45170
I20260809 06:45:48.449385 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30378
I20260809 06:45:48.467774 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.450808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.27.209.193:32973)
0809 06:45:48.451019 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.451025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.452221 (+ 1196us) server_negotiation.cc:407] Connection header received
0809 06:45:48.452408 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.452414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.452494 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.452622 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:48.453970 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.455133 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.456951 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.457673 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.457806 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.460714 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:48.460721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:48.460726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:48.460768 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:48.464544 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.465372 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.465379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.465382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.465691 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.466132 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.466137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.466141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.466506 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:48.466803 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.467080 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.467332 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 06:45:48.468655 30594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42120
W20260809 06:45:48.469448 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:45:48.469677 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30378
I20260809 06:45:48.502795 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30604
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:45:48.519428 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.503599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.27.209.194:35733)
0809 06:45:48.503780 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.503786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.503806 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:45:48.504006 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.504012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.504097 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.504214 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:48.505329 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.506474 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.507755 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.508464 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.508586 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.512570 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:48.512587 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:48.512608 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:48.512657 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:48.516528 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.517219 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.517230 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.517234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.517397 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.517753 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.517758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.517762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.518235 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:48.518559 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.518733 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.519031 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:45:48.520213 30819 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57148
W20260809 06:45:48.520828 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30604: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:45:48.521011 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30604
I20260809 06:45:48.544658 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30837
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[30197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257947, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:45:48.561436 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:48.545403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.27.209.195:40517)
0809 06:45:48.545947 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:48.545952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:48.545968 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:45:48.546026 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:48.546030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:48.546103 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:48.546200 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:48.547324 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.548258 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.549995 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:48.550506 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:48.550621 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:48.555766 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:48.555777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:48.555791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:48.555827 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:48.558808 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.559581 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.559589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.559591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.559753 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:48.560099 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:48.560103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:48.560105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:48.560432 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:48.560690 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:48.560920 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 06:45:48.561071 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":295,"threads_started":1}
W20260809 06:45:48.562234 31040 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52166
W20260809 06:45:48.562932 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:45:48.563077 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30837
I20260809 06:45:48.586861 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30214
I20260809 06:45:48.588020 30328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:48.701488 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30214
2026-08-09T06:45:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4573 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[31175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[31175](info): set up 2 sockets
Aug 09 06:45:48 dist-test-slave-jpj1 krb5kdc[31175](info): commencing operation
krb5kdc: starting...
W20260809 06:45:49.883060 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:45:50 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:45:50.066897 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38537
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:50.387930 31194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:50.388206 31194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:50.388332 31194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:50.395325 31194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:45:50.395432 31194 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 06:45:50.395498 31194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:50.395604 31194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:50.395674 31194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:45:50.395759 31194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:45:50.405406 31194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:38537
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:50.406664 31194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:50.407964 31194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:50.419275 31200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:50.419883 31201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:50.419893 31204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:50.420347 31194 server_base.cc:1034] running on GCE node
I20260809 06:45:50.421099 31194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:50.422483 31194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:50.423707 31194 hybrid_clock.cc:648] HybridClock initialized: now 1786257950423644 us; error 39 us; skew 500 ppm
I20260809 06:45:50.424006 31194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:50 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:50.431103 31194 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:45:50.432631 31194 webserver.cc:460] Webserver started at http://127.27.209.254:46473/ using document root <none> and password file <none>
I20260809 06:45:50.433322 31194 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:50.433424 31194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:50.433743 31194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:50.436656 31194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "5e4c2c9d1a1c44f290900b6252f61e5f"
format_stamp: "Formatted at 2026-08-09 06:45:50 on dist-test-slave-jpj1"
server_key: "8df180e6e45338111dcfe03f01afd5c6"
server_key_iv: "c78e5d95121434cee62efc2fb403411f"
server_key_version: "encryptionkey@0"
I20260809 06:45:50.437399 31194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "5e4c2c9d1a1c44f290900b6252f61e5f"
format_stamp: "Formatted at 2026-08-09 06:45:50 on dist-test-slave-jpj1"
server_key: "8df180e6e45338111dcfe03f01afd5c6"
server_key_iv: "c78e5d95121434cee62efc2fb403411f"
server_key_version: "encryptionkey@0"
I20260809 06:45:50.442337 31194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:45:50.445693 31213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:50.446939 31194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:45:50.447093 31194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "5e4c2c9d1a1c44f290900b6252f61e5f"
format_stamp: "Formatted at 2026-08-09 06:45:50 on dist-test-slave-jpj1"
server_key: "8df180e6e45338111dcfe03f01afd5c6"
server_key_iv: "c78e5d95121434cee62efc2fb403411f"
server_key_version: "encryptionkey@0"
I20260809 06:45:50.447289 31194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:50.477924 31194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:50.481659 31194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:50.481897 31194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:50.501392 31194 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:38537
I20260809 06:45:50.501410 31307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:38537 every 8 connection(s)
I20260809 06:45:50.502652 31194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:45:50.503079 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 31194
I20260809 06:45:50.503266 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:45:50.504102 28487 external_mini_cluster.cc:1428] Setting key a7dbaaccce79123b37e5ca152b85ffec
I20260809 06:45:50.507283 31309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:45:50 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:50.519753 31309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Bootstrap starting.
I20260809 06:45:50.522994 31309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:50.524260 31309 log.cc:824] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:50.526669 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:50.506100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.27.209.254:38537)
0809 06:45:50.506973 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:50.506989 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:50.507040 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:45:50.509247 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:50.509274 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:50.509770 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:50.510177 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:50.511381 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:50.513433 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:50.514595 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:50.515602 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:50.515787 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:50.519821 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:50.519832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:50.519849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:50.519897 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:50.522676 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:50.523483 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:50.523489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:50.523495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:50.523703 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:50.524092 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:50.524095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:50.524097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:50.524610 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:50.524890 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:50.525370 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 06:45:50.525737 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":347,"threads_started":1}
I20260809 06:45:50.526973 31309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: No bootstrap required, opened a new log
I20260809 06:45:50.534942 31309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38537 } }
I20260809 06:45:50.535269 31309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:50.535364 31309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4c2c9d1a1c44f290900b6252f61e5f, State: Initialized, Role: FOLLOWER
I20260809 06:45:50.535960 31309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38537 } }
I20260809 06:45:50.536231 31309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:45:50.536363 31309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:45:50.536526 31309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:50.538653 31309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38537 } }
I20260809 06:45:50.539124 31309 leader_election.cc:302] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e4c2c9d1a1c44f290900b6252f61e5f; no voters:
I20260809 06:45:50.539799 31309 leader_election.cc:288] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:45:50.539916 31317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:50.540597 31317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [term 1 LEADER]: Becoming Leader. State: Replica: 5e4c2c9d1a1c44f290900b6252f61e5f, State: Running, Role: LEADER
I20260809 06:45:50.541070 31317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38537 } }
I20260809 06:45:50.541716 31309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:45:50.543840 31318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38537 } } }
I20260809 06:45:50.544306 31318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:50.545004 31319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e4c2c9d1a1c44f290900b6252f61e5f. Latest consensus state: current_term: 1 leader_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4c2c9d1a1c44f290900b6252f61e5f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38537 } } }
I20260809 06:45:50.545293 31319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:50.548791 31327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:45:50.554144 31327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:45:50.562896 31327 catalog_manager.cc:1347] Generated new cluster ID: e12bcc0f1dde4ddf93cb2a0ac6ea1c54
I20260809 06:45:50.562992 31327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:45:50.582549 31327 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:45:50.584358 31327 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:45:50.600266 31327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Generated new TSK 0
I20260809 06:45:50.601059 31327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:45:50.658730 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38537
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:50.998453 31347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:50.998740 31347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:50.998888 31347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:51.006335 31347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:51.006448 31347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:51.006585 31347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 06:45:51.006656 31347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:45:51.006763 31347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:45:51.016531 31347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:38537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:51.017863 31347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:51.019162 31347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:51.032114 31354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:51.032146 31355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:51.032070 31357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:51.033123 31347 server_base.cc:1034] running on GCE node
I20260809 06:45:51.033742 31347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:51.034655 31347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:51.035842 31347 hybrid_clock.cc:648] HybridClock initialized: now 1786257951035798 us; error 35 us; skew 500 ppm
I20260809 06:45:51.036136 31347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:51 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257951, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:51.039076 31347 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:45:51.040515 31347 webserver.cc:460] Webserver started at http://127.27.209.193:45945/ using document root <none> and password file <none>
I20260809 06:45:51.041177 31347 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:51.041296 31347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:51.041600 31347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:51.044672 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "9a7ec41025104bda998f8087e8479cc0"
format_stamp: "Formatted at 2026-08-09 06:45:51 on dist-test-slave-jpj1"
server_key: "bafbbcb411700d768db82bd25f0713fd"
server_key_iv: "65ab0aafc27f4ee2a4bb304a5536c96d"
server_key_version: "encryptionkey@0"
I20260809 06:45:51.045470 31347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "9a7ec41025104bda998f8087e8479cc0"
format_stamp: "Formatted at 2026-08-09 06:45:51 on dist-test-slave-jpj1"
server_key: "bafbbcb411700d768db82bd25f0713fd"
server_key_iv: "65ab0aafc27f4ee2a4bb304a5536c96d"
server_key_version: "encryptionkey@0"
I20260809 06:45:51.050493 31347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:45:51.053884 31371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:51.055094 31347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:45:51.055248 31347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "9a7ec41025104bda998f8087e8479cc0"
format_stamp: "Formatted at 2026-08-09 06:45:51 on dist-test-slave-jpj1"
server_key: "bafbbcb411700d768db82bd25f0713fd"
server_key_iv: "65ab0aafc27f4ee2a4bb304a5536c96d"
server_key_version: "encryptionkey@0"
I20260809 06:45:51.055431 31347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:51.085274 31347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:51.089874 31347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:51.090159 31347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:51.092867 31347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:51.092972 31347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:51.093114 31347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:51.093194 31347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:45:51 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257951, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:51.115482 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.093304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:34285 (local address 127.27.209.254:38537)
0809 06:45:51.095806 (+ 2502us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:51.095811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:51.095829 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:45:51.098018 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:51.098024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:51.098131 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:51.098242 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:51.100272 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.101237 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.103698 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.104238 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.104366 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.108946 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:51.108954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:51.108959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:51.109004 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:51.112280 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.113172 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.113178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.113182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.113342 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.113788 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.113792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.113795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.114056 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:51.114189 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:51.114902 (+ 713us) server_negotiation.cc:299] Negotiation successful
0809 06:45:51.115121 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2382,"thread_start_us":2282,"threads_started":1}
I20260809 06:45:51.116688 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.093969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.27.209.193:34285)
0809 06:45:51.094917 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:51.094966 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:51.097556 (+ 2590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:51.098513 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:51.098521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:51.099274 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:51.100094 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.100114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.101464 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.101469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.103556 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.103573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.104572 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.104579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.104921 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.105680 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:51.105711 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:51.108698 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:51.112422 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.112432 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.112448 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.113015 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.113465 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.113473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.113476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.113657 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.114181 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:51.114189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:51.114511 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 06:45:51.114736 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 06:45:51.115225 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":385,"threads_started":1}
I20260809 06:45:51.135008 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.121856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:41351 (local address 127.27.209.254:38537)
0809 06:45:51.121999 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:51.122003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:51.122661 (+ 658us) server_negotiation.cc:407] Connection header received
0809 06:45:51.122830 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:51.122834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:51.122905 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:51.123000 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:51.124555 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.125625 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.127338 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.127891 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.127998 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.129733 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:51.129738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:51.129741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:51.129772 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:51.132356 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.133165 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.133168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.133171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.133288 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.133651 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.133656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.133660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.133932 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:51.134082 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:51.134588 (+ 506us) server_negotiation.cc:299] Negotiation successful
0809 06:45:51.134748 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:45:51.135356 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.121981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.27.209.193:41351)
0809 06:45:51.122566 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:51.122582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:51.122708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:51.123055 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:51.123060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:51.123854 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:51.124346 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.124364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.125772 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.125781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.127200 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.127213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.128102 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.128113 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.128268 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.128852 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:51.128881 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:51.129540 (+ 659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:51.132498 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.132504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.132508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.133012 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.133408 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.133414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.133418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.133543 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.134071 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:51.134076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:51.134321 (+ 245us) client_negotiation.cc:769] Sending connection context
0809 06:45:51.134503 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 06:45:51.134850 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":325,"threads_started":1}
I20260809 06:45:51.146947 31347 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:38347
I20260809 06:45:51.147022 31548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:38347 every 8 connection(s)
I20260809 06:45:51.148196 31347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:45:51.149271 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 31347
I20260809 06:45:51.149480 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:45:51.150179 28487 external_mini_cluster.cc:1428] Setting key 90d1969e3b5a275ca79201f8752d39d7
I20260809 06:45:51.150380 31550 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38537
I20260809 06:45:51.150626 31550 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:51.151302 31550 heartbeater.cc:511] Master 127.27.209.254:38537 requested a full tablet report, sending...
I20260809 06:45:51.153354 31239 ts_manager.cc:194] Registered new tserver with Master: 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347)
I20260809 06:45:51.154881 31239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:34285
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:45:51.193128 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38537
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:51.514403 31555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:51.514678 31555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:51.514830 31555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:51.521938 31555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:51.522053 31555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:51.522176 31555 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 06:45:51.522243 31555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:45:51.522344 31555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:45:51.531739 31555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:38537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:51.532943 31555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:51.534190 31555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:51.546463 31567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:51.546562 31562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:51.546527 31563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:51.547037 31555 server_base.cc:1034] running on GCE node
I20260809 06:45:51.547699 31555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:51.548576 31555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:51.549767 31555 hybrid_clock.cc:648] HybridClock initialized: now 1786257951549729 us; error 35 us; skew 500 ppm
I20260809 06:45:51.550057 31555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:51 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257951, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:51.553248 31555 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:45:51.554663 31555 webserver.cc:460] Webserver started at http://127.27.209.194:39385/ using document root <none> and password file <none>
I20260809 06:45:51.555299 31555 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:51.555414 31555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:51.555751 31555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:51.558699 31555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "7cfab1a657dd4f129a56307e4eb233f2"
format_stamp: "Formatted at 2026-08-09 06:45:51 on dist-test-slave-jpj1"
server_key: "80fba0690ed0326ffe145b2641035080"
server_key_iv: "1215831b995113eb6e1c300d3b9df218"
server_key_version: "encryptionkey@0"
I20260809 06:45:51.559417 31555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "7cfab1a657dd4f129a56307e4eb233f2"
format_stamp: "Formatted at 2026-08-09 06:45:51 on dist-test-slave-jpj1"
server_key: "80fba0690ed0326ffe145b2641035080"
server_key_iv: "1215831b995113eb6e1c300d3b9df218"
server_key_version: "encryptionkey@0"
I20260809 06:45:51.564184 31555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 06:45:51.567378 31577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:51.568696 31555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:45:51.568840 31555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "7cfab1a657dd4f129a56307e4eb233f2"
format_stamp: "Formatted at 2026-08-09 06:45:51 on dist-test-slave-jpj1"
server_key: "80fba0690ed0326ffe145b2641035080"
server_key_iv: "1215831b995113eb6e1c300d3b9df218"
server_key_version: "encryptionkey@0"
I20260809 06:45:51.569010 31555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:51.609766 31555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:51.614042 31555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:51.614331 31555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:51.617134 31555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:51.617249 31555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:51.617375 31555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:51.617493 31555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:45:51 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257951, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:51.640127 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.617643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:38293 (local address 127.27.209.254:38537)
0809 06:45:51.617875 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:51.617882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:51.619348 (+ 1466us) server_negotiation.cc:407] Connection header received
0809 06:45:51.621411 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:51.621416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:51.621500 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:51.621588 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:51.623590 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.624618 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.627082 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.627716 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.627854 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.633594 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:51.633601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:51.633606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:51.633646 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:51.636414 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.637274 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.637281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.637285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.637446 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.637982 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.637988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.637992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.638306 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:51.638460 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:51.639522 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0809 06:45:51.639776 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260809 06:45:51.641630 31592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.618371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.27.209.194:38293)
0809 06:45:51.619174 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:51.619232 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:51.621126 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:51.621841 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:51.621850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:51.622519 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:51.623312 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.623347 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.624808 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.624812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.626942 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.626957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.627997 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.628007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.628309 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.629053 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:51.629093 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:51.633370 (+ 4277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:51.636548 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.636558 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.636577 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.637124 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.637593 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.637599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.637602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.637864 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.638456 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:51.638464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:51.639018 (+ 554us) client_negotiation.cc:769] Sending connection context
0809 06:45:51.639421 (+ 403us) client_negotiation.cc:240] Negotiation successful
0809 06:45:51.640039 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":312,"threads_started":1}
I20260809 06:45:51.661506 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.647758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:38159 (local address 127.27.209.254:38537)
0809 06:45:51.647972 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:51.647977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:51.648652 (+ 675us) server_negotiation.cc:407] Connection header received
0809 06:45:51.648867 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:51.648872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:51.648947 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:51.649035 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:51.650406 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.651453 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.652827 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.653357 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.653482 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.655882 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:51.655887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:51.655890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:51.655919 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:51.658353 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.659194 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.659198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.659200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.659334 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:51.659816 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:51.659821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:51.659824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:51.660183 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:51.660317 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:51.660819 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 06:45:51.661058 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:45:51.661511 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:51.647827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.27.209.194:38159)
0809 06:45:51.648551 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:51.648573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:51.648725 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:51.649085 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:51.649090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:51.649724 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:51.650208 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.650222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.651610 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.651616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.652678 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:51.652690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:51.653544 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:51.653550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:51.653752 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:51.654409 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:51.654434 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:51.655687 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:51.658467 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.658473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.658478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.659064 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.659466 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:51.659472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:51.659475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:51.659648 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:51.660296 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:51.660300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:51.660499 (+ 199us) client_negotiation.cc:769] Sending connection context
0809 06:45:51.660641 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 06:45:51.660988 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":405,"threads_started":1}
I20260809 06:45:51.669234 31555 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:42729
I20260809 06:45:51.669284 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:42729 every 8 connection(s)
I20260809 06:45:51.670465 31555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:45:51.671834 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 31555
I20260809 06:45:51.671985 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:45:51.672489 28487 external_mini_cluster.cc:1428] Setting key aad18a4324fa1845d43e710c6b297aaa
I20260809 06:45:51.672382 31761 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38537
I20260809 06:45:51.673449 31761 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:51.674196 31761 heartbeater.cc:511] Master 127.27.209.254:38537 requested a full tablet report, sending...
I20260809 06:45:51.675603 31239 ts_manager.cc:194] Registered new tserver with Master: 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:51.676409 31239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:38293
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:45:51.712666 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38537
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:52.029379 31768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:52.029718 31768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:52.029874 31768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:52.037082 31768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:52.037214 31768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:52.037338 31768 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 06:45:52.037405 31768 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:45:52.037511 31768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:45:52.047722 31768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:38537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:52.049145 31768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:52.050567 31768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:52.063760 31778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:52.063735 31775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:52.063827 31776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:52.064666 31768 server_base.cc:1034] running on GCE node
I20260809 06:45:52.065310 31768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:52.066236 31768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:52.067472 31768 hybrid_clock.cc:648] HybridClock initialized: now 1786257952067416 us; error 43 us; skew 500 ppm
I20260809 06:45:52.067848 31768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:52 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:52.071354 31768 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:45:52.072947 31768 webserver.cc:460] Webserver started at http://127.27.209.195:34051/ using document root <none> and password file <none>
I20260809 06:45:52.073695 31768 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:52.073814 31768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:52.074139 31768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:52.077284 31768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "60ddbd525cc24be6830d4ec7f4f10fc7"
format_stamp: "Formatted at 2026-08-09 06:45:52 on dist-test-slave-jpj1"
server_key: "075b8037751e5c0c291bfef0f19f5a6d"
server_key_iv: "e278977202d6ecd6d919fc5cec4e81bd"
server_key_version: "encryptionkey@0"
I20260809 06:45:52.078059 31768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "60ddbd525cc24be6830d4ec7f4f10fc7"
format_stamp: "Formatted at 2026-08-09 06:45:52 on dist-test-slave-jpj1"
server_key: "075b8037751e5c0c291bfef0f19f5a6d"
server_key_iv: "e278977202d6ecd6d919fc5cec4e81bd"
server_key_version: "encryptionkey@0"
I20260809 06:45:52.083449 31768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:45:52.087021 31786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:52.088757 31768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:45:52.088935 31768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "60ddbd525cc24be6830d4ec7f4f10fc7"
format_stamp: "Formatted at 2026-08-09 06:45:52 on dist-test-slave-jpj1"
server_key: "075b8037751e5c0c291bfef0f19f5a6d"
server_key_iv: "e278977202d6ecd6d919fc5cec4e81bd"
server_key_version: "encryptionkey@0"
I20260809 06:45:52.089186 31768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:52.119524 31768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:52.123915 31768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:52.124207 31768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:52.126905 31768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:52.127024 31768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:52.127139 31768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:52.127226 31768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:45:52 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:52.149658 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.127754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:54195 (local address 127.27.209.254:38537)
0809 06:45:52.127952 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.127958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.129372 (+ 1414us) server_negotiation.cc:407] Connection header received
0809 06:45:52.131485 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.131491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.131602 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.131695 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:52.133803 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.134758 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.137326 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.137752 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.137864 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.142187 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:52.142194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:52.142198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:52.142240 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:52.145651 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.146713 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.146719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.146722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.146908 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.147479 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.147483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.147487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.147880 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:52.148035 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.149033 (+ 998us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.149273 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:45:52.151257 31799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.128329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.27.209.195:54195)
0809 06:45:52.129169 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.129223 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.131194 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.131981 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.131994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.132739 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:52.133570 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.133592 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.135009 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.135018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.137099 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.137129 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.138212 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.138220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.138545 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.139349 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:52.139376 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:52.141947 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:52.145853 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:52.145863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:52.145884 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:52.146526 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:52.147046 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:52.147051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:52.147055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:52.147327 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:52.148038 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:52.148048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:52.148521 (+ 473us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.148862 (+ 341us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.149464 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":355,"threads_started":1}
I20260809 06:45:52.157892 31550 heartbeater.cc:503] Master 127.27.209.254:38537 was elected leader, sending a full tablet report...
I20260809 06:45:52.171463 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.157145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:58033 (local address 127.27.209.254:38537)
0809 06:45:52.157328 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.157333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.158553 (+ 1220us) server_negotiation.cc:407] Connection header received
0809 06:45:52.158753 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.158768 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.158846 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.158938 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:52.160494 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.161205 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.162822 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.163211 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.163318 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.165670 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:52.165675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:52.165678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:52.165715 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:52.168626 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.169459 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.169464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.169468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.169636 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.169991 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.170015 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.170018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.170339 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:52.170487 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.170882 (+ 395us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.171097 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:45:52.171599 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.157271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.27.209.195:58033)
0809 06:45:52.158457 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.158476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.158599 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.159016 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.159020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.159645 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:52.160352 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.160369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.161349 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.161355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.162680 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.162693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.163391 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.163398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.163590 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.164315 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:52.164332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:52.165419 (+ 1087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:52.168783 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:52.168787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:52.168790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:52.169343 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:52.169751 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:52.169756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:52.169760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:52.169889 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:52.170455 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:52.170459 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:52.170636 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.170769 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.171138 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":406,"threads_started":1}
I20260809 06:45:52.182694 31768 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:37185
I20260809 06:45:52.182767 31958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:37185 every 8 connection(s)
I20260809 06:45:52.184038 31768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:45:52.186046 31960 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38537
I20260809 06:45:52.186262 31960 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:52.186959 31960 heartbeater.cc:511] Master 127.27.209.254:38537 requested a full tablet report, sending...
I20260809 06:45:52.188377 31239 ts_manager.cc:194] Registered new tserver with Master: 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185)
I20260809 06:45:52.189203 31239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:54195
I20260809 06:45:52.190676 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 31768
I20260809 06:45:52.190819 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:45:52.191251 28487 external_mini_cluster.cc:1428] Setting key 2d71aa1d5f3476260331d4dadbb57047
I20260809 06:45:52.193982 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:45:52 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:45:52 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:52.240702 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.227972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.27.209.254:38537)
0809 06:45:52.228137 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.228142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.228411 (+ 269us) server_negotiation.cc:407] Connection header received
0809 06:45:52.228577 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.228583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.228681 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.228777 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:52.230061 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.230779 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.231968 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.232392 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.232486 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.236016 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:52.236021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:52.236025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:52.236056 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:52.238513 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.239070 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.239074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.239076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.239201 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.239511 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.239514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.239516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.239819 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:52.240004 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.240231 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.240417 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:45:52.244935 31239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:45:52.251950 31239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:52.275308 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.264066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37185 (local address 127.0.0.1:47936)
0809 06:45:52.264871 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.264922 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.265138 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.266875 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.266883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.266925 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.267645 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.267659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.269932 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.269942 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.272420 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.272429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.274299 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.274306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.274491 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.274548 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.274694 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.274905 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":43,"spinlock_wait_cycles":1280,"thread_start_us":389,"threads_started":1}
I20260809 06:45:52.276119 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.266269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.27.209.193:38347)
0809 06:45:52.266783 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.266793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.268038 (+ 1245us) server_negotiation.cc:407] Connection header received
0809 06:45:52.268149 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.268157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.268401 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.268623 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.269143 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.270384 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.272612 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.274086 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.274278 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.274464 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.275661 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.275837 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":237,"threads_started":1}
I20260809 06:45:52.276207 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.264265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.27.209.195:37185)
0809 06:45:52.266225 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.266234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.266268 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:45:52.266368 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.266376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.266646 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.267027 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.268228 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.269791 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.272598 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.274118 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.274296 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.274464 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.275805 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.275935 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1752,"thread_start_us":1559,"threads_started":1}
I20260809 06:45:52.275504 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.265318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38347 (local address 127.0.0.1:56676)
0809 06:45:52.267837 (+ 2519us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.267858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.267981 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.268606 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.268611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.268637 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.268972 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.268983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.270568 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.270572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.272413 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.272427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.274652 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.274658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.274842 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.274874 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.274963 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.275094 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2399,"mutex_wait_us":3,"thread_start_us":2198,"threads_started":1}
I20260809 06:45:52.277262 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.264543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42729 (local address 127.0.0.1:47062)
0809 06:45:52.265731 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.265750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.265863 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.266875 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.266883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.266934 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.268047 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.268060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.269647 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.269653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.272395 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.272412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.276477 (+ 4065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.276491 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.276669 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.276701 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.276790 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.276914 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":24,"spinlock_wait_cycles":18816,"thread_start_us":941,"threads_started":1}
I20260809 06:45:52.277323 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.265720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.27.209.194:42729)
0809 06:45:52.266327 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.266334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.266356 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:45:52.266444 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.266450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.266621 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.267127 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.268284 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.269519 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.274565 (+ 5046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.275980 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.276177 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.276350 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.276870 (+ 520us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.277037 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":330,"threads_started":1}
I20260809 06:45:52.280036 31861 tablet_service.cc:1467] Processing CreateTablet for tablet da4525f866d14643a0bcc62f859fbd77 (DEFAULT_TABLE table=test-table [id=74077e90a38440239478273689c7c628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:52.280288 31452 tablet_service.cc:1467] Processing CreateTablet for tablet da4525f866d14643a0bcc62f859fbd77 (DEFAULT_TABLE table=test-table [id=74077e90a38440239478273689c7c628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:52.280692 31656 tablet_service.cc:1467] Processing CreateTablet for tablet da4525f866d14643a0bcc62f859fbd77 (DEFAULT_TABLE table=test-table [id=74077e90a38440239478273689c7c628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:52.281293 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4525f866d14643a0bcc62f859fbd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:52.281711 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4525f866d14643a0bcc62f859fbd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:52.281950 31656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da4525f866d14643a0bcc62f859fbd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:52.292548 31986 tablet_bootstrap.cc:492] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Bootstrap starting.
I20260809 06:45:52.296016 31988 tablet_bootstrap.cc:492] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Bootstrap starting.
I20260809 06:45:52.296687 31986 tablet_bootstrap.cc:654] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:52.298043 31987 tablet_bootstrap.cc:492] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Bootstrap starting.
I20260809 06:45:52.298120 31986 log.cc:824] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:52.299456 31988 tablet_bootstrap.cc:654] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:52.300689 31988 log.cc:824] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:52.300956 31986 tablet_bootstrap.cc:492] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: No bootstrap required, opened a new log
I20260809 06:45:52.301227 31986 ts_tablet_manager.cc:1397] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:45:52.301435 31987 tablet_bootstrap.cc:654] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:52.302428 31987 log.cc:824] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:52.303526 31988 tablet_bootstrap.cc:492] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: No bootstrap required, opened a new log
I20260809 06:45:52.303828 31988 ts_tablet_manager.cc:1397] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:45:52.307026 31987 tablet_bootstrap.cc:492] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: No bootstrap required, opened a new log
I20260809 06:45:52.307300 31987 ts_tablet_manager.cc:1397] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:45:52.309484 31986 raft_consensus.cc:348] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.309870 31986 raft_consensus.cc:374] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:52.309971 31986 raft_consensus.cc:729] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Initialized, Role: FOLLOWER
I20260809 06:45:52.310544 31986 consensus_queue.cc:260] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.311662 31960 heartbeater.cc:503] Master 127.27.209.254:38537 was elected leader, sending a full tablet report...
I20260809 06:45:52.312191 31986 ts_tablet_manager.cc:1428] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260809 06:45:52.314090 31988 raft_consensus.cc:348] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.314524 31988 raft_consensus.cc:374] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:52.314635 31988 raft_consensus.cc:729] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Initialized, Role: FOLLOWER
I20260809 06:45:52.315251 31988 consensus_queue.cc:260] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.318645 31987 raft_consensus.cc:348] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.319146 31987 raft_consensus.cc:374] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:52.319252 31987 raft_consensus.cc:729] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfab1a657dd4f129a56307e4eb233f2, State: Initialized, Role: FOLLOWER
I20260809 06:45:52.321029 31988 ts_tablet_manager.cc:1428] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260809 06:45:52.319931 31987 consensus_queue.cc:260] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.322360 31761 heartbeater.cc:503] Master 127.27.209.254:38537 was elected leader, sending a full tablet report...
I20260809 06:45:52.322819 31987 ts_tablet_manager.cc:1428] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260809 06:45:52.332582 31995 raft_consensus.cc:482] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:52.332864 31995 raft_consensus.cc:504] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.334429 31995 leader_election.cc:288] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185), 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:52.341017 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.334919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:42483 (local address 127.27.209.194:42729)
0809 06:45:52.335128 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.335134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.335618 (+ 484us) server_negotiation.cc:407] Connection header received
0809 06:45:52.335792 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.335798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.335876 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.335943 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.336598 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.337450 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.339300 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.340085 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.340214 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.340288 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.340643 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.340774 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:45:52.341046 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.334743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:50557 (local address 127.27.209.195:37185)
0809 06:45:52.334947 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.334954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.335750 (+ 796us) server_negotiation.cc:407] Connection header received
0809 06:45:52.335904 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.335908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.335983 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.336091 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.336706 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.337411 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.339313 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.340128 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.340292 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.340362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.340674 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.340807 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:45:52.341238 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.334720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42729 (local address 127.27.209.193:42483)
0809 06:45:52.335473 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.335496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.335659 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.335980 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.335984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.336016 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.336443 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.336455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.337592 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.337596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.339148 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.339161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.340313 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.340320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.340483 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.340523 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.340594 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.340712 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":77,"thread_start_us":404,"threads_started":1}
I20260809 06:45:52.341491 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.334803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37185 (local address 127.27.209.193:50557)
0809 06:45:52.335639 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.335656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.335777 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.336127 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.336130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.336163 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.336539 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.336550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.337560 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.337566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.339142 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.339156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.340359 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.340364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.340519 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.340543 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.340632 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.340753 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":43,"thread_start_us":529,"threads_started":1}
I20260809 06:45:52.342245 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4525f866d14643a0bcc62f859fbd77" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2" is_pre_election: true
I20260809 06:45:52.342428 31895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4525f866d14643a0bcc62f859fbd77" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" is_pre_election: true
I20260809 06:45:52.342778 31690 raft_consensus.cc:2454] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 0.
I20260809 06:45:52.342941 31895 raft_consensus.cc:2454] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 0.
I20260809 06:45:52.343431 31374 leader_election.cc:302] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cfab1a657dd4f129a56307e4eb233f2, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:52.343859 31995 raft_consensus.cc:2781] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:52.343994 31995 raft_consensus.cc:482] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:52.344120 31995 raft_consensus.cc:3037] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:52.346750 31995 raft_consensus.cc:504] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.347371 31995 leader_election.cc:288] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185), 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:52.347787 31895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4525f866d14643a0bcc62f859fbd77" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7"
I20260809 06:45:52.347862 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da4525f866d14643a0bcc62f859fbd77" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2"
I20260809 06:45:52.347966 31895 raft_consensus.cc:3037] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:52.348042 31690 raft_consensus.cc:3037] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:52.350276 31895 raft_consensus.cc:2454] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 1.
I20260809 06:45:52.350255 31690 raft_consensus.cc:2454] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 1.
I20260809 06:45:52.350658 31376 leader_election.cc:302] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:52.350971 31995 raft_consensus.cc:2781] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:52.351442 31995 raft_consensus.cc:686] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Running, Role: LEADER
I20260809 06:45:52.351933 31995 consensus_queue.cc:237] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } }
I20260809 06:45:52.356385 31238 catalog_manager.cc:5495] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "9a7ec41025104bda998f8087e8479cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:45:52 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
W20260809 06:45:52.400522 31551 tablet.cc:2369] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:52.403285 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.389994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38537 (local address 127.0.0.1:49180)
0809 06:45:52.390595 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.390611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.390756 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.391091 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.391096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.391881 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:52.392154 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.392164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.393173 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.393178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.394222 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.394235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.394940 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.394948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.395092 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.395767 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:52.395783 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:52.397896 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:52.401015 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:52.401027 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:52.401035 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:52.401543 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:52.401899 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:52.401902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:52.401904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:52.402033 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:52.402504 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:52.402509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:52.402652 (+ 143us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.402782 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.402967 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":340,"threads_started":1}
I20260809 06:45:52.403366 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.389973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.27.209.254:38537)
0809 06:45:52.390147 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.390152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.390702 (+ 550us) server_negotiation.cc:407] Connection header received
0809 06:45:52.390881 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.390886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.390957 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.391038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:52.392304 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.393052 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.394357 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.394768 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.394877 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.398076 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:52.398083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:52.398087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:52.398118 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:52.400887 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.401661 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.401667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.401671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.401806 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:52.402123 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:52.402126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:52.402128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:52.402399 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:52.402508 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.402911 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.403085 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:45:52.407999 31238 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:49180:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 06:45:52.409030 31238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:52.415174 31452 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef9316a6fb440158d4a2557307bd675 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba903b30cdec4a1b8939660788d2034b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:45:52.415439 31656 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef9316a6fb440158d4a2557307bd675 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba903b30cdec4a1b8939660788d2034b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:45:52.415517 31861 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef9316a6fb440158d4a2557307bd675 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba903b30cdec4a1b8939660788d2034b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:45:52.416252 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef9316a6fb440158d4a2557307bd675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:52.416368 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef9316a6fb440158d4a2557307bd675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:52.416419 31656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef9316a6fb440158d4a2557307bd675. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:52.422366 31987 tablet_bootstrap.cc:492] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: Bootstrap starting.
W20260809 06:45:52.422405 31763 tablet.cc:2369] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:52.423269 31988 tablet_bootstrap.cc:492] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: Bootstrap starting.
I20260809 06:45:52.423420 31986 tablet_bootstrap.cc:492] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: Bootstrap starting.
I20260809 06:45:52.424110 31987 tablet_bootstrap.cc:654] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:52.424618 31988 tablet_bootstrap.cc:654] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:52.425114 31986 tablet_bootstrap.cc:654] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:52.426990 31988 tablet_bootstrap.cc:492] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: No bootstrap required, opened a new log
I20260809 06:45:52.427192 31988 ts_tablet_manager.cc:1397] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:45:52.427914 31986 tablet_bootstrap.cc:492] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: No bootstrap required, opened a new log
I20260809 06:45:52.428061 31987 tablet_bootstrap.cc:492] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: No bootstrap required, opened a new log
I20260809 06:45:52.428067 31986 ts_tablet_manager.cc:1397] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 06:45:52.428042 31988 raft_consensus.cc:348] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.428220 31987 ts_tablet_manager.cc:1397] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 06:45:52.428283 31988 raft_consensus.cc:374] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:52.428390 31988 raft_consensus.cc:729] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Initialized, Role: FOLLOWER
I20260809 06:45:52.428619 31988 consensus_queue.cc:260] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.429244 31988 ts_tablet_manager.cc:1428] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:45:52.429080 31995 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.429414 31995 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:52.429312 31986 raft_consensus.cc:348] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.429564 31986 raft_consensus.cc:374] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:52.429678 31986 raft_consensus.cc:729] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Initialized, Role: FOLLOWER
I20260809 06:45:52.428923 31987 raft_consensus.cc:348] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.429873 31987 raft_consensus.cc:374] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:52.429984 31987 raft_consensus.cc:729] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfab1a657dd4f129a56307e4eb233f2, State: Initialized, Role: FOLLOWER
I20260809 06:45:52.429903 31986 consensus_queue.cc:260] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.430467 31986 ts_tablet_manager.cc:1428] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 06:45:52.430385 31994 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.430757 31994 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:52.431411 31987 consensus_queue.cc:260] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.432052 31987 ts_tablet_manager.cc:1428] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 06:45:52.431947 31996 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.432341 31996 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260809 06:45:52.435500 31961 tablet.cc:2369] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:45:52.761281 31996 raft_consensus.cc:482] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:52.761531 31996 raft_consensus.cc:504] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.763154 31996 leader_election.cc:288] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347), 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185)
I20260809 06:45:52.770138 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.763510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:48091 (local address 127.27.209.193:38347)
0809 06:45:52.763723 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.763730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.764239 (+ 509us) server_negotiation.cc:407] Connection header received
0809 06:45:52.764509 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.764514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.764604 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.764658 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.765495 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.766329 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.768306 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.769147 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.769286 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.769341 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.769732 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.769898 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:45:52.770182 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.764354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:46337 (local address 127.27.209.195:37185)
0809 06:45:52.764529 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.764534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.764871 (+ 337us) server_negotiation.cc:407] Connection header received
0809 06:45:52.765016 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.765021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.765096 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.765194 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:52.765859 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.766617 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.768535 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.769307 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.769449 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.769520 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.769843 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.769971 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:45:52.770337 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.763469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38347 (local address 127.27.209.194:48091)
0809 06:45:52.764101 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.764124 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.764309 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.764829 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.764834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.764897 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.765307 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.765322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.766473 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.766478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.768142 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.768155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.769374 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.769381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.769553 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.769596 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.769666 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.769806 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":32,"spinlock_wait_cycles":47872,"thread_start_us":373,"threads_started":1}
I20260809 06:45:52.770380 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.764192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37185 (local address 127.27.209.194:46337)
0809 06:45:52.764762 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.764776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.764893 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.765252 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.765256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.765283 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:52.765675 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.765687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.766806 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.766813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.768381 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.768395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.769512 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.769516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.769659 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.769696 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.769787 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.769909 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":19,"thread_start_us":283,"threads_started":1}
I20260809 06:45:52.771072 31895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef9316a6fb440158d4a2557307bd675" candidate_uuid: "7cfab1a657dd4f129a56307e4eb233f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" is_pre_election: true
I20260809 06:45:52.771405 31895 raft_consensus.cc:2454] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cfab1a657dd4f129a56307e4eb233f2 in term 0.
I20260809 06:45:52.771405 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef9316a6fb440158d4a2557307bd675" candidate_uuid: "7cfab1a657dd4f129a56307e4eb233f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7ec41025104bda998f8087e8479cc0" is_pre_election: true
I20260809 06:45:52.771936 31582 leader_election.cc:302] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 7cfab1a657dd4f129a56307e4eb233f2; no voters:
I20260809 06:45:52.772006 31492 raft_consensus.cc:2454] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cfab1a657dd4f129a56307e4eb233f2 in term 0.
I20260809 06:45:52.772356 31996 raft_consensus.cc:2781] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:52.772496 31996 raft_consensus.cc:482] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:52.772601 31996 raft_consensus.cc:3037] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:52.774757 31996 raft_consensus.cc:504] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.775311 31996 leader_election.cc:288] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [CANDIDATE]: Term 1 election: Requested vote from peers 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347), 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185)
I20260809 06:45:52.775745 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef9316a6fb440158d4a2557307bd675" candidate_uuid: "7cfab1a657dd4f129a56307e4eb233f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7ec41025104bda998f8087e8479cc0"
I20260809 06:45:52.775835 31895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef9316a6fb440158d4a2557307bd675" candidate_uuid: "7cfab1a657dd4f129a56307e4eb233f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7"
I20260809 06:45:52.775941 31492 raft_consensus.cc:3037] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:52.776041 31895 raft_consensus.cc:3037] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:52.778834 31492 raft_consensus.cc:2454] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cfab1a657dd4f129a56307e4eb233f2 in term 1.
I20260809 06:45:52.778829 31895 raft_consensus.cc:2454] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cfab1a657dd4f129a56307e4eb233f2 in term 1.
I20260809 06:45:52.779232 31582 leader_election.cc:302] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cfab1a657dd4f129a56307e4eb233f2, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:52.779512 31996 raft_consensus.cc:2781] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:52.779934 31996 raft_consensus.cc:686] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 LEADER]: Becoming Leader. State: Replica: 7cfab1a657dd4f129a56307e4eb233f2, State: Running, Role: LEADER
I20260809 06:45:52.780395 31996 consensus_queue.cc:237] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:52.783840 32018 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: TxnStatusTablet state changed. Reason: New leader 7cfab1a657dd4f129a56307e4eb233f2. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.784246 32018 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:45:52.784863 32020 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 06:45:52.785095 32020 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 06:45:52.785140 31238 catalog_manager.cc:5495] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194). New cstate: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } health_report { overall_health: UNKNOWN } } }
I20260809 06:45:52.793115 31995 consensus_queue.cc:1035] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:52.801208 31999 consensus_queue.cc:1035] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:52.884218 32008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.878640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42729 (local address 127.0.0.1:47076)
0809 06:45:52.878858 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.878875 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.879017 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.879690 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.879695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.879706 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:52.879894 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.879904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.880957 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.880963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.881948 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.881961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.882644 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.882650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.882788 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.882845 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.883687 (+ 842us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.883708 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.883787 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.883941 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 06:45:52.884264 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.878780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.27.209.194:42729)
0809 06:45:52.879306 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.879310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.879328 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:45:52.879409 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.879413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.879492 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.879658 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:52.880023 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.880836 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.882081 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.882497 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.882607 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.883013 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.883581 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.883708 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.883838 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.883916 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":255,"threads_started":1}
I20260809 06:45:52.891769 31895 raft_consensus.cc:1261] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Refusing update from remote peer 7cfab1a657dd4f129a56307e4eb233f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:52.891906 31492 raft_consensus.cc:1261] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Refusing update from remote peer 7cfab1a657dd4f129a56307e4eb233f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:52.892540 32018 consensus_queue.cc:1035] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:52.892877 31996 consensus_queue.cc:1035] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:52.899245 31995 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: TxnStatusTablet state changed. Reason: New leader 7cfab1a657dd4f129a56307e4eb233f2. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.899523 31995 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:52.901881 32024 mvcc.cc:205] Tried to move back new op lower bound from 7316512575035527168 to 7316512574592008192. Current Snapshot: MvccSnapshot[applied={T|T < 7316512575035527168}]
I20260809 06:45:52.903609 32023 mvcc.cc:205] Tried to move back new op lower bound from 7316512575035527168 to 7316512574592008192. Current Snapshot: MvccSnapshot[applied={T|T < 7316512575035527168}]
I20260809 06:45:52.904577 32018 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.904832 32018 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:45:52.914803 31994 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: TxnStatusTablet state changed. Reason: New leader 7cfab1a657dd4f129a56307e4eb233f2. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.915097 31994 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:52.916532 31995 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.916803 31995 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 9a7ec41025104bda998f8087e8479cc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:52.918424 31996 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.918715 31996 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 7cfab1a657dd4f129a56307e4eb233f2: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:45:52.920148 32027 mvcc.cc:205] Tried to move back new op lower bound from 7316512575035527168 to 7316512574592008192. Current Snapshot: MvccSnapshot[applied={T|T < 7316512575035527168}]
I20260809 06:45:52.922863 31994 tablet_replica.cc:439] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cfab1a657dd4f129a56307e4eb233f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } } }
I20260809 06:45:52.923092 31994 tablet_replica.cc:442] T 1ef9316a6fb440158d4a2557307bd675 P 60ddbd525cc24be6830d4ec7f4f10fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:45:52.932085 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.925424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.27.209.193:38347)
0809 06:45:52.925595 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.925602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.926169 (+ 567us) server_negotiation.cc:407] Connection header received
0809 06:45:52.926405 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.926411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.926511 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.926665 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:52.927081 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.928217 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.929553 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.930210 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.930345 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.930714 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.931421 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.931593 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.931676 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.931762 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:45:52.948122 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.942057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42729 (local address 127.27.209.193:58125)
0809 06:45:52.942759 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.942795 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.942911 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.943237 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.943241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.943250 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:52.943442 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.943449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.944910 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.944914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.945983 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.945995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.946957 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.946962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.947095 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.947151 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.947617 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.947659 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.947729 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.947833 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":430,"threads_started":1}
I20260809 06:45:52.948124 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.942270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:58125 (local address 127.27.209.194:42729)
0809 06:45:52.942479 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.942486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.942873 (+ 387us) server_negotiation.cc:407] Connection header received
0809 06:45:52.943020 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.943025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.943114 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.943221 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:52.943608 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.944780 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.946121 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.946759 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.946905 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.947347 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.947480 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.947592 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.947777 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.947860 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:45:52.988880 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.982110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:44771 (local address 127.27.209.193:38347)
0809 06:45:52.982285 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:52.982291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:52.982883 (+ 592us) server_negotiation.cc:407] Connection header received
0809 06:45:52.983099 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:52.983104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:52.983179 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:52.983254 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:52.983999 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.985074 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.987098 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.987670 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.987799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.988262 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.988358 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.988426 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:52.988576 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 06:45:52.988666 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:45:52.989142 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:52.982120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38347 (local address 127.27.209.194:44771)
0809 06:45:52.982789 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:52.982805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:52.982945 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:52.983417 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:52.983421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:52.983433 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:52.983808 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.983821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.985268 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.985273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.986969 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:52.986985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:52.987841 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:52.987847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:52.988026 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:52.988096 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:52.988463 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:52.988478 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 06:45:52.988565 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 06:45:52.988703 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":326,"threads_started":1}
I20260809 06:45:53.027194 31238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:53.027501 31238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:53.031575 31238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Sending DeleteTablet for 3 replicas of tablet da4525f866d14643a0bcc62f859fbd77
I20260809 06:45:53.032831 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet da4525f866d14643a0bcc62f859fbd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:53 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56676
I20260809 06:45:53.032938 31656 tablet_service.cc:1514] Processing DeleteTablet for tablet da4525f866d14643a0bcc62f859fbd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:53 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47062
I20260809 06:45:53.033818 31861 tablet_service.cc:1514] Processing DeleteTablet for tablet da4525f866d14643a0bcc62f859fbd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:53 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47936
I20260809 06:45:53.036192 31239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:45:53.036900 31239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:53.045312 31655 tablet_service.cc:1467] Processing CreateTablet for tablet f9a9bcc6606b406da7ea8676ff47d8c5 (DEFAULT_TABLE table=test-table [id=af35db3c86704f7fb0e86acca0da8701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:53.046010 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a9bcc6606b406da7ea8676ff47d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:53.047942 32055 tablet_replica.cc:330] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: stopping tablet replica
I20260809 06:45:53.048421 32055 raft_consensus.cc:2229] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:53.048462 31449 tablet_service.cc:1467] Processing CreateTablet for tablet f9a9bcc6606b406da7ea8676ff47d8c5 (DEFAULT_TABLE table=test-table [id=af35db3c86704f7fb0e86acca0da8701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:53.048761 32055 raft_consensus.cc:2258] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:53.049036 31449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a9bcc6606b406da7ea8676ff47d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:53.050866 32055 ts_tablet_manager.cc:1905] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:53.052263 31857 tablet_service.cc:1467] Processing CreateTablet for tablet f9a9bcc6606b406da7ea8676ff47d8c5 (DEFAULT_TABLE table=test-table [id=af35db3c86704f7fb0e86acca0da8701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:53.052772 31857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a9bcc6606b406da7ea8676ff47d8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:53.058259 32055 ts_tablet_manager.cc:1918] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:53.058502 32055 log.cc:1196] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/da4525f866d14643a0bcc62f859fbd77
I20260809 06:45:53.059209 32055 ts_tablet_manager.cc:1939] T da4525f866d14643a0bcc62f859fbd77 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting consensus metadata
I20260809 06:45:53.061794 31214 catalog_manager.cc:4841] TS 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729): tablet da4525f866d14643a0bcc62f859fbd77 (table test-table [id=74077e90a38440239478273689c7c628]) successfully deleted
I20260809 06:45:53.061990 32058 tablet_bootstrap.cc:492] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Bootstrap starting.
I20260809 06:45:53.063928 32058 tablet_bootstrap.cc:654] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:53.084364 32065 tablet_bootstrap.cc:492] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Bootstrap starting.
I20260809 06:45:53.085019 32062 tablet_bootstrap.cc:492] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Bootstrap starting.
I20260809 06:45:53.085144 32058 tablet_bootstrap.cc:492] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: No bootstrap required, opened a new log
I20260809 06:45:53.085296 32058 ts_tablet_manager.cc:1397] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.011s
I20260809 06:45:53.086156 32058 raft_consensus.cc:348] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.086413 32058 raft_consensus.cc:374] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:53.086510 32058 raft_consensus.cc:729] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfab1a657dd4f129a56307e4eb233f2, State: Initialized, Role: FOLLOWER
I20260809 06:45:53.086784 32058 consensus_queue.cc:260] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.087414 32058 ts_tablet_manager.cc:1428] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 06:45:53.088981 32065 tablet_bootstrap.cc:654] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:53.089514 32056 tablet_replica.cc:330] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: stopping tablet replica
I20260809 06:45:53.091058 32056 raft_consensus.cc:2229] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:53.092012 32056 raft_consensus.cc:2258] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:53.090595 32062 tablet_bootstrap.cc:654] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:53.091672 32052 tablet_replica.cc:330] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: stopping tablet replica
I20260809 06:45:53.094689 32052 raft_consensus.cc:2229] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:45:53.096210 32052 raft_consensus.cc:2258] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:53.096743 32056 ts_tablet_manager.cc:1905] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:53.097697 32065 tablet_bootstrap.cc:492] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: No bootstrap required, opened a new log
I20260809 06:45:53.097855 32065 ts_tablet_manager.cc:1397] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260809 06:45:53.098698 32065 raft_consensus.cc:348] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.098955 32065 raft_consensus.cc:374] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:53.099056 32065 raft_consensus.cc:729] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Initialized, Role: FOLLOWER
I20260809 06:45:53.099324 32065 consensus_queue.cc:260] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.101305 32052 ts_tablet_manager.cc:1905] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:53.103929 32065 ts_tablet_manager.cc:1428] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260809 06:45:53.104766 32056 ts_tablet_manager.cc:1918] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:53.104992 32056 log.cc:1196] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/da4525f866d14643a0bcc62f859fbd77
I20260809 06:45:53.105605 32056 ts_tablet_manager.cc:1939] T da4525f866d14643a0bcc62f859fbd77 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting consensus metadata
I20260809 06:45:53.107507 31218 catalog_manager.cc:4841] TS 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185): tablet da4525f866d14643a0bcc62f859fbd77 (table test-table [id=74077e90a38440239478273689c7c628]) successfully deleted
I20260809 06:45:53.108811 32062 tablet_bootstrap.cc:492] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: No bootstrap required, opened a new log
I20260809 06:45:53.108966 32062 ts_tablet_manager.cc:1397] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20260809 06:45:53.109789 32062 raft_consensus.cc:348] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.110039 32062 raft_consensus.cc:374] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:53.110141 32062 raft_consensus.cc:729] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Initialized, Role: FOLLOWER
I20260809 06:45:53.110409 32062 consensus_queue.cc:260] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.111104 32062 ts_tablet_manager.cc:1428] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 06:45:53.113054 32052 ts_tablet_manager.cc:1918] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:53.113301 32052 log.cc:1196] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/da4525f866d14643a0bcc62f859fbd77
I20260809 06:45:53.114861 32052 ts_tablet_manager.cc:1939] T da4525f866d14643a0bcc62f859fbd77 P 9a7ec41025104bda998f8087e8479cc0: Deleting consensus metadata
I20260809 06:45:53.116612 31218 catalog_manager.cc:4841] TS 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347): tablet da4525f866d14643a0bcc62f859fbd77 (table test-table [id=74077e90a38440239478273689c7c628]) successfully deleted
I20260809 06:45:53.377310 31994 raft_consensus.cc:482] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:53.377512 31994 raft_consensus.cc:504] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.379034 31994 leader_election.cc:288] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347), 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:53.386873 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.379576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:53791 (local address 127.27.209.194:42729)
0809 06:45:53.379783 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:53.379790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:53.380633 (+ 843us) server_negotiation.cc:407] Connection header received
0809 06:45:53.380956 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:53.380962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:53.381037 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:53.381119 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:53.381621 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.382680 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.385235 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.386007 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.386164 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.386255 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:53.386496 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 06:45:53.386637 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:45:53.386935 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.379289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42729 (local address 127.27.209.195:53791)
0809 06:45:53.380521 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:53.380543 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:53.380764 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:53.381141 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:53.381144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:53.381167 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:53.381441 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.381454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.382844 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.382849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.385034 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.385050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.386177 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.386183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.386345 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.386371 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 06:45:53.386453 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 06:45:53.386584 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":13,"thread_start_us":983,"threads_started":1}
I20260809 06:45:53.387717 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a9bcc6606b406da7ea8676ff47d8c5" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2" is_pre_election: true
I20260809 06:45:53.387905 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.379946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:38745 (local address 127.27.209.193:38347)
0809 06:45:53.380133 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:53.380139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:53.380974 (+ 835us) server_negotiation.cc:407] Connection header received
0809 06:45:53.381136 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:53.381141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:53.381239 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:53.381339 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:45:53.381918 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.382659 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.385217 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.386632 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.386828 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.386907 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:53.387464 (+ 557us) server_negotiation.cc:299] Negotiation successful
0809 06:45:53.387615 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:45:53.387964 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.379292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38347 (local address 127.27.209.195:38745)
0809 06:45:53.380865 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:53.380883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:53.381004 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:53.381372 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:53.381376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:53.381426 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:45:53.381763 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.381771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.382782 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.382788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.385039 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.385050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.387063 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.387071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.387290 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.387327 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:45:53.387429 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 06:45:53.387585 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1448,"mutex_wait_us":133,"thread_start_us":1210,"threads_started":1}
I20260809 06:45:53.387986 31690 raft_consensus.cc:2454] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 0.
I20260809 06:45:53.388670 31788 leader_election.cc:302] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 7cfab1a657dd4f129a56307e4eb233f2; no voters:
I20260809 06:45:53.388654 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a9bcc6606b406da7ea8676ff47d8c5" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7ec41025104bda998f8087e8479cc0" is_pre_election: true
I20260809 06:45:53.388998 31490 raft_consensus.cc:2454] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 0.
I20260809 06:45:53.389060 31994 raft_consensus.cc:2781] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:53.389189 31994 raft_consensus.cc:482] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:53.389290 31994 raft_consensus.cc:3037] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:53.391490 31994 raft_consensus.cc:504] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.392084 31994 leader_election.cc:288] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 election: Requested vote from peers 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347), 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:53.392560 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a9bcc6606b406da7ea8676ff47d8c5" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7ec41025104bda998f8087e8479cc0"
I20260809 06:45:53.392656 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a9bcc6606b406da7ea8676ff47d8c5" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2"
I20260809 06:45:53.392793 31490 raft_consensus.cc:3037] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:53.392890 31690 raft_consensus.cc:3037] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:53.395768 31690 raft_consensus.cc:2454] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 1.
I20260809 06:45:53.396034 31490 raft_consensus.cc:2454] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 1.
I20260809 06:45:53.396247 31788 leader_election.cc:302] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 7cfab1a657dd4f129a56307e4eb233f2; no voters:
I20260809 06:45:53.396605 31994 raft_consensus.cc:2781] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:53.397089 31994 raft_consensus.cc:686] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Running, Role: LEADER
I20260809 06:45:53.397594 31994 consensus_queue.cc:237] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:53.401628 31239 catalog_manager.cc:5495] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195). New cstate: current_term: 1 leader_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } health_report { overall_health: HEALTHY } } }
I20260809 06:45:53.457118 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.450907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.27.209.195:37185)
0809 06:45:53.451565 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:53.451571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:53.451833 (+ 262us) server_negotiation.cc:407] Connection header received
0809 06:45:53.452065 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:53.452069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:53.452152 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:53.452303 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:53.452730 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.453543 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.454912 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.455332 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.455440 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.455897 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:53.456474 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:53.456609 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:53.456718 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 06:45:53.456799 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":341,"threads_started":1}
I20260809 06:45:53.472671 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.466986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:34799 (local address 127.27.209.194:42729)
0809 06:45:53.467129 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:53.467133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:53.467518 (+ 385us) server_negotiation.cc:407] Connection header received
0809 06:45:53.467721 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:53.467725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:53.467788 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:53.467860 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:53.468369 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.469022 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.471006 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.471436 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.471577 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.472031 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:53.472150 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:53.472224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:53.472362 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 06:45:53.472427 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 06:45:53.472883 32083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.466798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42729 (local address 127.27.209.195:34799)
0809 06:45:53.467420 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:53.467440 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:53.467609 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:53.467885 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:53.467890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:53.467903 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:53.468225 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.468235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.469130 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.469135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.470869 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.470885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.471618 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.471624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.471800 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.471868 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:53.472251 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:53.472263 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 06:45:53.472349 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 06:45:53.472500 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":339,"threads_started":1}
I20260809 06:45:53.480718 31690 raft_consensus.cc:1261] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Refusing update from remote peer 60ddbd525cc24be6830d4ec7f4f10fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:53.480751 31490 raft_consensus.cc:1261] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Refusing update from remote peer 60ddbd525cc24be6830d4ec7f4f10fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:53.481374 32076 consensus_queue.cc:1035] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:53.481863 31994 consensus_queue.cc:1035] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:53.518353 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.501520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:42213 (local address 127.27.209.195:37185)
0809 06:45:53.501845 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:53.501852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:53.513588 (+ 11736us) server_negotiation.cc:407] Connection header received
0809 06:45:53.513758 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:53.513763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:53.513846 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:53.513930 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:53.514331 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.515336 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.516780 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.517188 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.517295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.517649 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:53.517746 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:53.517819 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:53.517970 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 06:45:53.518064 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260809 06:45:53.518419 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:53.501225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37185 (local address 127.27.209.194:42213)
0809 06:45:53.513481 (+ 12256us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:53.513497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:53.513620 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:53.513948 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:53.513953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:53.513965 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:45:53.514169 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.514179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.515594 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.515599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.516642 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:53.516656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:53.517334 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:53.517340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:53.517471 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:53.517508 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:45:53.517846 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:45:53.517857 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 06:45:53.517942 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 06:45:53.518095 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12136,"thread_start_us":12476,"threads_started":1}
W20260809 06:45:53.519452 32027 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 06:45:55.113862 31239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:55.114080 31239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:55.118021 31239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Sending DeleteTablet for 3 replicas of tablet f9a9bcc6606b406da7ea8676ff47d8c5
I20260809 06:45:55.118732 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a9bcc6606b406da7ea8676ff47d8c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56676
I20260809 06:45:55.118831 31655 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a9bcc6606b406da7ea8676ff47d8c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47062
I20260809 06:45:55.118873 31861 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a9bcc6606b406da7ea8676ff47d8c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47936
I20260809 06:45:55.128722 32131 tablet_replica.cc:330] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: stopping tablet replica
I20260809 06:45:55.129097 32131 raft_consensus.cc:2229] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:55.129431 32131 raft_consensus.cc:2258] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:55.130072 32132 tablet_replica.cc:330] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: stopping tablet replica
I20260809 06:45:55.130420 32127 tablet_replica.cc:330] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: stopping tablet replica
I20260809 06:45:55.130631 32127 raft_consensus.cc:2229] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:55.130882 32127 raft_consensus.cc:2258] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:55.130903 32131 ts_tablet_manager.cc:1905] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:55.131708 32132 raft_consensus.cc:2229] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:45:55.132198 32132 raft_consensus.cc:2258] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:55.132995 32132 ts_tablet_manager.cc:1905] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:55.133333 32127 ts_tablet_manager.cc:1905] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:55.137909 32132 ts_tablet_manager.cc:1918] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:45:55.137919 32131 ts_tablet_manager.cc:1918] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:45:55.138038 32132 log.cc:1196] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/f9a9bcc6606b406da7ea8676ff47d8c5
I20260809 06:45:55.138052 32131 log.cc:1196] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/f9a9bcc6606b406da7ea8676ff47d8c5
I20260809 06:45:55.138572 32131 ts_tablet_manager.cc:1939] T f9a9bcc6606b406da7ea8676ff47d8c5 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting consensus metadata
I20260809 06:45:55.139652 32127 ts_tablet_manager.cc:1918] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:45:55.139801 32127 log.cc:1196] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/f9a9bcc6606b406da7ea8676ff47d8c5
I20260809 06:45:55.139835 31214 catalog_manager.cc:4841] TS 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729): tablet f9a9bcc6606b406da7ea8676ff47d8c5 (table test-table [id=af35db3c86704f7fb0e86acca0da8701]) successfully deleted
I20260809 06:45:55.140224 32127 ts_tablet_manager.cc:1939] T f9a9bcc6606b406da7ea8676ff47d8c5 P 9a7ec41025104bda998f8087e8479cc0: Deleting consensus metadata
I20260809 06:45:55.141161 31218 catalog_manager.cc:4841] TS 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347): tablet f9a9bcc6606b406da7ea8676ff47d8c5 (table test-table [id=af35db3c86704f7fb0e86acca0da8701]) successfully deleted
I20260809 06:45:55.141654 32132 ts_tablet_manager.cc:1939] T f9a9bcc6606b406da7ea8676ff47d8c5 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting consensus metadata
I20260809 06:45:55.142575 31218 catalog_manager.cc:4841] TS 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185): tablet f9a9bcc6606b406da7ea8676ff47d8c5 (table test-table [id=af35db3c86704f7fb0e86acca0da8701]) successfully deleted
Aug 09 06:45:55 dist-test-slave-jpj1 krb5kdc[31175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:45:55 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:55.182245 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:55.168806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.27.209.254:38537)
0809 06:45:55.169426 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:55.169432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:55.169457 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:45:55.169647 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:55.169651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:55.169714 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:55.169807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:55.171154 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:55.172006 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:55.173284 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:55.173705 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:55.173814 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:55.177690 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:55.177697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:55.177701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:55.177735 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:55.180190 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:55.180799 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:55.180802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:55.180805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:55.180930 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:55.181201 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:55.181204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:55.181206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:55.181451 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:55.181632 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:55.181808 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 06:45:55.181969 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":317,"threads_started":1}
I20260809 06:45:55.185774 31239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:45:55.186434 31239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:55.192826 31452 tablet_service.cc:1467] Processing CreateTablet for tablet 4444a6efea1049359de6a403ed37832d (DEFAULT_TABLE table=test-table [id=a162bd38d3dc413f8acc7a85e07f2961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:55.193040 31861 tablet_service.cc:1467] Processing CreateTablet for tablet 4444a6efea1049359de6a403ed37832d (DEFAULT_TABLE table=test-table [id=a162bd38d3dc413f8acc7a85e07f2961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:55.193231 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4444a6efea1049359de6a403ed37832d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:55.193482 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4444a6efea1049359de6a403ed37832d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:55.196184 31655 tablet_service.cc:1467] Processing CreateTablet for tablet 4444a6efea1049359de6a403ed37832d (DEFAULT_TABLE table=test-table [id=a162bd38d3dc413f8acc7a85e07f2961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:55.196645 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4444a6efea1049359de6a403ed37832d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:55.206624 32147 tablet_bootstrap.cc:492] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Bootstrap starting.
I20260809 06:45:55.206646 32149 tablet_bootstrap.cc:492] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Bootstrap starting.
I20260809 06:45:55.207391 32148 tablet_bootstrap.cc:492] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Bootstrap starting.
I20260809 06:45:55.208109 32147 tablet_bootstrap.cc:654] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:55.208563 32149 tablet_bootstrap.cc:654] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:55.209067 32148 tablet_bootstrap.cc:654] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:55.210538 32147 tablet_bootstrap.cc:492] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: No bootstrap required, opened a new log
I20260809 06:45:55.210698 32147 ts_tablet_manager.cc:1397] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:45:55.210852 32149 tablet_bootstrap.cc:492] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: No bootstrap required, opened a new log
I20260809 06:45:55.210981 32149 ts_tablet_manager.cc:1397] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:45:55.211205 32148 tablet_bootstrap.cc:492] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: No bootstrap required, opened a new log
I20260809 06:45:55.211347 32148 ts_tablet_manager.cc:1397] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 06:45:55.211467 32147 raft_consensus.cc:348] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.211740 32147 raft_consensus.cc:374] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:55.211671 32149 raft_consensus.cc:348] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.211843 32147 raft_consensus.cc:729] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Initialized, Role: FOLLOWER
I20260809 06:45:55.211889 32149 raft_consensus.cc:374] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:55.211984 32149 raft_consensus.cc:729] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfab1a657dd4f129a56307e4eb233f2, State: Initialized, Role: FOLLOWER
I20260809 06:45:55.212112 32147 consensus_queue.cc:260] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.212112 32148 raft_consensus.cc:348] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.212213 32149 consensus_queue.cc:260] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.212352 32148 raft_consensus.cc:374] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:55.212440 32148 raft_consensus.cc:729] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Initialized, Role: FOLLOWER
I20260809 06:45:55.212715 32149 ts_tablet_manager.cc:1428] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:45:55.212680 32148 consensus_queue.cc:260] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.212872 32147 ts_tablet_manager.cc:1428] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:45:55.213513 32148 ts_tablet_manager.cc:1428] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 06:45:55.252414 32096 raft_consensus.cc:482] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:55.252633 32096 raft_consensus.cc:504] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.253255 32096 leader_election.cc:288] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347), 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:55.253975 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4444a6efea1049359de6a403ed37832d" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2" is_pre_election: true
I20260809 06:45:55.253995 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4444a6efea1049359de6a403ed37832d" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7ec41025104bda998f8087e8479cc0" is_pre_election: true
I20260809 06:45:55.254267 31690 raft_consensus.cc:2454] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 0.
I20260809 06:45:55.254244 31490 raft_consensus.cc:2454] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 0.
I20260809 06:45:55.254777 31788 leader_election.cc:302] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 7cfab1a657dd4f129a56307e4eb233f2, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:55.255056 32096 raft_consensus.cc:2781] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:55.255182 32096 raft_consensus.cc:482] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:55.255271 32096 raft_consensus.cc:3037] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:55.257334 32096 raft_consensus.cc:504] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.257851 32096 leader_election.cc:288] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 election: Requested vote from peers 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347), 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729)
I20260809 06:45:55.258374 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4444a6efea1049359de6a403ed37832d" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a7ec41025104bda998f8087e8479cc0"
I20260809 06:45:55.258385 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4444a6efea1049359de6a403ed37832d" candidate_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2"
I20260809 06:45:55.258564 31490 raft_consensus.cc:3037] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:55.258617 31690 raft_consensus.cc:3037] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:55.261034 31690 raft_consensus.cc:2454] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 1.
I20260809 06:45:55.261053 31490 raft_consensus.cc:2454] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ddbd525cc24be6830d4ec7f4f10fc7 in term 1.
I20260809 06:45:55.261431 31788 leader_election.cc:302] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 7cfab1a657dd4f129a56307e4eb233f2, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:55.261715 32096 raft_consensus.cc:2781] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:55.261849 32096 raft_consensus.cc:686] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Running, Role: LEADER
I20260809 06:45:55.262130 32096 consensus_queue.cc:237] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.264515 31239 catalog_manager.cc:5495] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195). New cstate: current_term: 1 leader_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } health_report { overall_health: HEALTHY } } }
I20260809 06:45:55.292619 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:55.288014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.27.209.195:37185)
0809 06:45:55.288517 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:55.288522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:55.288540 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:45:55.288590 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:55.288594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:55.288651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:55.288730 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:55.289055 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:55.289835 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:55.291089 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:55.291488 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:55.291626 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:55.292028 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:55.292132 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:55.292194 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:55.292284 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 06:45:55.292344 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
I20260809 06:45:55.300544 31490 raft_consensus.cc:1261] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Refusing update from remote peer 60ddbd525cc24be6830d4ec7f4f10fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:55.300629 31690 raft_consensus.cc:1261] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Refusing update from remote peer 60ddbd525cc24be6830d4ec7f4f10fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:55.301190 32118 consensus_queue.cc:1035] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:55.301544 32096 consensus_queue.cc:1035] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:55.342885 31237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:55.343158 31237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:55.345973 31237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Sending DeleteTablet for 3 replicas of tablet 4444a6efea1049359de6a403ed37832d
I20260809 06:45:55.346652 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet 4444a6efea1049359de6a403ed37832d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56676
I20260809 06:45:55.346690 31861 tablet_service.cc:1514] Processing DeleteTablet for tablet 4444a6efea1049359de6a403ed37832d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47936
I20260809 06:45:55.346768 31655 tablet_service.cc:1514] Processing DeleteTablet for tablet 4444a6efea1049359de6a403ed37832d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47062
I20260809 06:45:55.346969 32127 tablet_replica.cc:330] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: stopping tablet replica
I20260809 06:45:55.347028 32131 tablet_replica.cc:330] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: stopping tablet replica
I20260809 06:45:55.347165 32127 raft_consensus.cc:2229] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:55.347245 32131 raft_consensus.cc:2229] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:55.347348 32127 raft_consensus.cc:2258] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:55.347425 32131 raft_consensus.cc:2258] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:55.347633 32132 tablet_replica.cc:330] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: stopping tablet replica
I20260809 06:45:55.347842 32132 raft_consensus.cc:2229] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:45:55.347757 31237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 06:45:55.348162 32132 raft_consensus.cc:2258] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:55.348433 32131 ts_tablet_manager.cc:1905] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 06:45:55.348418 31237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:45:55.349416 32132 ts_tablet_manager.cc:1905] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:55.354743 32131 ts_tablet_manager.cc:1918] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:55.354910 32131 log.cc:1196] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/4444a6efea1049359de6a403ed37832d
I20260809 06:45:55.355376 32131 ts_tablet_manager.cc:1939] T 4444a6efea1049359de6a403ed37832d P 7cfab1a657dd4f129a56307e4eb233f2: Deleting consensus metadata
I20260809 06:45:55.355450 32132 ts_tablet_manager.cc:1918] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:55.355643 32132 log.cc:1196] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/4444a6efea1049359de6a403ed37832d
I20260809 06:45:55.356063 32132 ts_tablet_manager.cc:1939] T 4444a6efea1049359de6a403ed37832d P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting consensus metadata
I20260809 06:45:55.356153 31452 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8db737f5cc418fbac449824e35e158 (DEFAULT_TABLE table=test-table [id=28d3717eec3d434bb62b5e0ddd5c6c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:55.356627 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8db737f5cc418fbac449824e35e158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:55.356979 32127 ts_tablet_manager.cc:1905] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:55.357031 31218 catalog_manager.cc:4841] TS 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185): tablet 4444a6efea1049359de6a403ed37832d (table test-table [id=a162bd38d3dc413f8acc7a85e07f2961]) successfully deleted
I20260809 06:45:55.361727 32148 tablet_bootstrap.cc:492] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Bootstrap starting.
I20260809 06:45:55.361986 31861 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8db737f5cc418fbac449824e35e158 (DEFAULT_TABLE table=test-table [id=28d3717eec3d434bb62b5e0ddd5c6c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:55.362202 32127 ts_tablet_manager.cc:1918] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:45:55.362315 32127 log.cc:1196] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/4444a6efea1049359de6a403ed37832d
I20260809 06:45:55.362370 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8db737f5cc418fbac449824e35e158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:55.362639 32127 ts_tablet_manager.cc:1939] T 4444a6efea1049359de6a403ed37832d P 9a7ec41025104bda998f8087e8479cc0: Deleting consensus metadata
I20260809 06:45:55.363381 32148 tablet_bootstrap.cc:654] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:55.363397 31218 catalog_manager.cc:4841] TS 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347): tablet 4444a6efea1049359de6a403ed37832d (table test-table [id=a162bd38d3dc413f8acc7a85e07f2961]) successfully deleted
I20260809 06:45:55.365392 32148 tablet_bootstrap.cc:492] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: No bootstrap required, opened a new log
I20260809 06:45:55.365557 32148 ts_tablet_manager.cc:1397] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260809 06:45:55.366254 32148 raft_consensus.cc:348] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.366503 32148 raft_consensus.cc:374] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:55.366588 32148 raft_consensus.cc:729] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Initialized, Role: FOLLOWER
I20260809 06:45:55.366837 32148 consensus_queue.cc:260] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.367365 32148 ts_tablet_manager.cc:1428] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:45:55.368054 31214 catalog_manager.cc:4841] TS 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729): tablet 4444a6efea1049359de6a403ed37832d (table test-table [id=a162bd38d3dc413f8acc7a85e07f2961]) successfully deleted
I20260809 06:45:55.368502 31655 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8db737f5cc418fbac449824e35e158 (DEFAULT_TABLE table=test-table [id=28d3717eec3d434bb62b5e0ddd5c6c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:45:55.368866 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8db737f5cc418fbac449824e35e158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:55.371855 32147 tablet_bootstrap.cc:492] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Bootstrap starting.
I20260809 06:45:55.373245 32147 tablet_bootstrap.cc:654] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:55.373739 32149 tablet_bootstrap.cc:492] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Bootstrap starting.
I20260809 06:45:55.375138 32147 tablet_bootstrap.cc:492] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: No bootstrap required, opened a new log
I20260809 06:45:55.375144 32149 tablet_bootstrap.cc:654] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:55.375286 32147 ts_tablet_manager.cc:1397] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260809 06:45:55.375927 32147 raft_consensus.cc:348] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.376190 32147 raft_consensus.cc:374] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:55.376278 32147 raft_consensus.cc:729] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ddbd525cc24be6830d4ec7f4f10fc7, State: Initialized, Role: FOLLOWER
I20260809 06:45:55.376518 32147 consensus_queue.cc:260] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.376998 32149 tablet_bootstrap.cc:492] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: No bootstrap required, opened a new log
I20260809 06:45:55.377146 32149 ts_tablet_manager.cc:1397] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 06:45:55.377198 32147 ts_tablet_manager.cc:1428] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:45:55.377831 32149 raft_consensus.cc:348] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.378068 32149 raft_consensus.cc:374] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:55.378167 32149 raft_consensus.cc:729] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cfab1a657dd4f129a56307e4eb233f2, State: Initialized, Role: FOLLOWER
I20260809 06:45:55.378402 32149 consensus_queue.cc:260] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.378892 32149 ts_tablet_manager.cc:1428] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:45:55.535346 32155 raft_consensus.cc:482] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:45:55.535609 32155 raft_consensus.cc:504] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.536412 32155 leader_election.cc:288] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729), 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185)
I20260809 06:45:55.536995 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8db737f5cc418fbac449824e35e158" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2" is_pre_election: true
I20260809 06:45:55.537061 31894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8db737f5cc418fbac449824e35e158" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" is_pre_election: true
I20260809 06:45:55.537269 31690 raft_consensus.cc:2454] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 0.
I20260809 06:45:55.537320 31894 raft_consensus.cc:2454] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 0.
I20260809 06:45:55.537669 31374 leader_election.cc:302] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cfab1a657dd4f129a56307e4eb233f2, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:55.537930 32155 raft_consensus.cc:2781] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:45:55.538050 32155 raft_consensus.cc:482] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:45:55.538139 32155 raft_consensus.cc:3037] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:55.540230 32155 raft_consensus.cc:504] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.540764 32155 leader_election.cc:288] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729), 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185)
I20260809 06:45:55.541199 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8db737f5cc418fbac449824e35e158" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cfab1a657dd4f129a56307e4eb233f2"
I20260809 06:45:55.541316 31894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8db737f5cc418fbac449824e35e158" candidate_uuid: "9a7ec41025104bda998f8087e8479cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7"
I20260809 06:45:55.541452 31690 raft_consensus.cc:3037] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:55.541514 31894 raft_consensus.cc:3037] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:55.544476 31690 raft_consensus.cc:2454] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 1.
I20260809 06:45:55.544507 31894 raft_consensus.cc:2454] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7ec41025104bda998f8087e8479cc0 in term 1.
I20260809 06:45:55.544936 31376 leader_election.cc:302] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60ddbd525cc24be6830d4ec7f4f10fc7, 7cfab1a657dd4f129a56307e4eb233f2, 9a7ec41025104bda998f8087e8479cc0; no voters:
I20260809 06:45:55.545225 32155 raft_consensus.cc:2781] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:55.545630 32155 raft_consensus.cc:686] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 1 LEADER]: Becoming Leader. State: Replica: 9a7ec41025104bda998f8087e8479cc0, State: Running, Role: LEADER
I20260809 06:45:55.545910 32155 consensus_queue.cc:237] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } }
I20260809 06:45:55.548688 31236 catalog_manager.cc:5495] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "9a7ec41025104bda998f8087e8479cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a7ec41025104bda998f8087e8479cc0" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 } health_report { overall_health: UNKNOWN } } }
I20260809 06:45:55.610852 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:55.606094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.27.209.193:38347)
0809 06:45:55.606640 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:55.606644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:55.606663 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:45:55.606713 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:55.606717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:55.606773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:55.606852 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:45:55.607188 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:55.608028 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:55.609356 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:55.609783 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:55.609887 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:55.610262 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:45:55.610362 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:45:55.610429 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:55.610521 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 06:45:55.610581 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":314,"threads_started":1}
I20260809 06:45:55.619062 31894 raft_consensus.cc:1261] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Refusing update from remote peer 9a7ec41025104bda998f8087e8479cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:55.619284 31690 raft_consensus.cc:1261] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Refusing update from remote peer 9a7ec41025104bda998f8087e8479cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:45:55.619884 32172 consensus_queue.cc:1035] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cfab1a657dd4f129a56307e4eb233f2" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:45:55.620198 32155 consensus_queue.cc:1035] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ddbd525cc24be6830d4ec7f4f10fc7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:45:55.637460 32103 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 06:45:57.233085 31237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:57.233342 31237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:45:57.237313 31237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e4c2c9d1a1c44f290900b6252f61e5f: Sending DeleteTablet for 3 replicas of tablet 8f8db737f5cc418fbac449824e35e158
I20260809 06:45:57.238029 31452 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8db737f5cc418fbac449824e35e158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:57 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56676
I20260809 06:45:57.238072 31655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8db737f5cc418fbac449824e35e158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:57 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47062
I20260809 06:45:57.238253 31861 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8db737f5cc418fbac449824e35e158 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:45:57 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47936
I20260809 06:45:57.238934 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31347
I20260809 06:45:57.251216 32208 tablet_replica.cc:330] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: stopping tablet replica
I20260809 06:45:57.251283 32209 tablet_replica.cc:330] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: stopping tablet replica
I20260809 06:45:57.251448 32208 raft_consensus.cc:2229] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:45:57.251729 32209 raft_consensus.cc:2229] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:57.251869 32208 raft_consensus.cc:2258] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:57.252228 32209 raft_consensus.cc:2258] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:57.252493 32210 tablet_replica.cc:330] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: stopping tablet replica
I20260809 06:45:57.252707 32210 raft_consensus.cc:2229] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:45:57.253013 32210 raft_consensus.cc:2258] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:45:57.253347 32209 ts_tablet_manager.cc:1905] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:57.254068 32210 ts_tablet_manager.cc:1905] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:57.254269 32208 ts_tablet_manager.cc:1905] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:45:57.260282 32209 ts_tablet_manager.cc:1918] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:45:57.260504 32209 log.cc:1196] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/8f8db737f5cc418fbac449824e35e158
I20260809 06:45:57.260843 32208 ts_tablet_manager.cc:1918] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 06:45:57.261010 32208 log.cc:1196] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/8f8db737f5cc418fbac449824e35e158
I20260809 06:45:57.261157 32209 ts_tablet_manager.cc:1939] T 8f8db737f5cc418fbac449824e35e158 P 7cfab1a657dd4f129a56307e4eb233f2: Deleting consensus metadata
I20260809 06:45:57.261493 32208 ts_tablet_manager.cc:1939] T 8f8db737f5cc418fbac449824e35e158 P 9a7ec41025104bda998f8087e8479cc0: Deleting consensus metadata
I20260809 06:45:57.261881 32210 ts_tablet_manager.cc:1918] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
Aug 09 06:45:57 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:45:57.262073 32210 log.cc:1196] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/8f8db737f5cc418fbac449824e35e158
I20260809 06:45:57.262542 31214 catalog_manager.cc:4841] TS 7cfab1a657dd4f129a56307e4eb233f2 (127.27.209.194:42729): tablet 8f8db737f5cc418fbac449824e35e158 (table test-table [id=28d3717eec3d434bb62b5e0ddd5c6c1b]) successfully deleted
I20260809 06:45:57.262562 31218 catalog_manager.cc:4841] TS 9a7ec41025104bda998f8087e8479cc0 (127.27.209.193:38347): tablet 8f8db737f5cc418fbac449824e35e158 (table test-table [id=28d3717eec3d434bb62b5e0ddd5c6c1b]) successfully deleted
I20260809 06:45:57.262804 32210 ts_tablet_manager.cc:1939] T 8f8db737f5cc418fbac449824e35e158 P 60ddbd525cc24be6830d4ec7f4f10fc7: Deleting consensus metadata
I20260809 06:45:57.263748 31218 catalog_manager.cc:4841] TS 60ddbd525cc24be6830d4ec7f4f10fc7 (127.27.209.195:37185): tablet 8f8db737f5cc418fbac449824e35e158 (table test-table [id=28d3717eec3d434bb62b5e0ddd5c6c1b]) successfully deleted
I20260809 06:45:57.268316 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:57.241472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.27.209.193:38347)
0809 06:45:57.249319 (+ 7847us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:57.249327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:57.251049 (+ 1722us) server_negotiation.cc:407] Connection header received
0809 06:45:57.251263 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:57.251269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:57.251348 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:57.251448 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:57.253627 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:57.254772 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:57.256901 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:57.257531 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:57.257655 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:57.263213 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:57.263229 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:57.263248 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:57.263284 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:57.265940 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:57.266555 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:57.266563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:57.266565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:57.266723 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:57.267026 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:57.267029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:57.267031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:57.267360 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:57.267634 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:57.267867 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 06:45:57.268033 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7693,"thread_start_us":7560,"threads_started":1}
W20260809 06:45:57.269032 31543 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56706
W20260809 06:45:57.269662 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:45:57.269862 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 31347
I20260809 06:45:57.302168 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31555
Aug 09 06:45:57 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:45:57.316457 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:57.302906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.27.209.194:42729)
0809 06:45:57.303427 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:57.303431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:57.303448 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:45:57.303503 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:57.303507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:57.303600 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:57.303694 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:57.304925 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:57.305665 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:57.306853 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:57.307231 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:57.307324 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:57.311307 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:57.311320 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:57.311336 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:57.311372 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:57.313989 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:57.314583 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:57.314590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:57.314592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:57.314722 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:57.315044 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:57.315047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:57.315049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:57.315388 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:57.315679 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:57.316018 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 06:45:57.316182 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
W20260809 06:45:57.317101 31750 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47086
W20260809 06:45:57.317675 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31555: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:45:57.317813 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 31555
I20260809 06:45:57.348470 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31768
Aug 09 06:45:57 dist-test-slave-jpj1 krb5kdc[31175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:45:57.362895 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:57.349156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.27.209.195:37185)
0809 06:45:57.349639 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:57.349644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:57.349661 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:45:57.349715 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:57.349719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:57.349783 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:57.349876 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:57.351224 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:57.352034 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:57.353318 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:57.353690 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:57.353781 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:57.357974 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:57.357990 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:57.358008 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:57.358041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:57.360615 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:57.361228 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:57.361236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:57.361238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:57.361376 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:57.361640 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:57.361643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:57.361645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:57.362000 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:57.362213 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:57.362421 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 06:45:57.362584 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":240,"threads_started":1}
W20260809 06:45:57.363592 31951 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47966
W20260809 06:45:57.364178 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31768: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:45:57.364339 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 31768
I20260809 06:45:57.393513 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31194
I20260809 06:45:57.394703 31302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:45:57.510711 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 31194
2026-08-09T06:45:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8812 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:45:57 dist-test-slave-jpj1 krb5kdc[32218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:45:57 dist-test-slave-jpj1 krb5kdc[32218](info): set up 2 sockets
Aug 09 06:45:57 dist-test-slave-jpj1 krb5kdc[32218](info): commencing operation
krb5kdc: starting...
W20260809 06:45:58.696064 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:45:58 dist-test-slave-jpj1 krb5kdc[32218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:45:58.881006 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44319
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:59.206179 32240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:59.206465 32240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:59.206597 32240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:59.213732 32240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:45:59.213858 32240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:59.213937 32240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:59.214018 32240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:45:59.214080 32240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:45:59.223855 32240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:59.225099 32240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:59.226336 32240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:59.237996 32248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:59.237958 32249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:59.238168 32240 server_base.cc:1034] running on GCE node
W20260809 06:45:59.238018 32253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:59.239411 32240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:59.240780 32240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:59.242002 32240 hybrid_clock.cc:648] HybridClock initialized: now 1786257959241953 us; error 33 us; skew 500 ppm
I20260809 06:45:59.242336 32240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:59 dist-test-slave-jpj1 krb5kdc[32218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:59.245441 32240 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:45:59.246865 32240 webserver.cc:460] Webserver started at http://127.27.209.254:36013/ using document root <none> and password file <none>
I20260809 06:45:59.247618 32240 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:59.247743 32240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:59.248057 32240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:59.251078 32240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "a793dac6be174f8c8f3ccee465a99e20"
format_stamp: "Formatted at 2026-08-09 06:45:59 on dist-test-slave-jpj1"
server_key: "d8ca4d476cf9ae4ee7de556ca1499e4e"
server_key_iv: "a76473c1f7c5506a7835c6326c7be32b"
server_key_version: "encryptionkey@0"
I20260809 06:45:59.251910 32240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "a793dac6be174f8c8f3ccee465a99e20"
format_stamp: "Formatted at 2026-08-09 06:45:59 on dist-test-slave-jpj1"
server_key: "d8ca4d476cf9ae4ee7de556ca1499e4e"
server_key_iv: "a76473c1f7c5506a7835c6326c7be32b"
server_key_version: "encryptionkey@0"
I20260809 06:45:59.257156 32240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:45:59.260710 32267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:59.262019 32240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:45:59.262197 32240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "a793dac6be174f8c8f3ccee465a99e20"
format_stamp: "Formatted at 2026-08-09 06:45:59 on dist-test-slave-jpj1"
server_key: "d8ca4d476cf9ae4ee7de556ca1499e4e"
server_key_iv: "a76473c1f7c5506a7835c6326c7be32b"
server_key_version: "encryptionkey@0"
I20260809 06:45:59.262398 32240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:59.290575 32240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:59.294924 32240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:59.295197 32240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:59.312630 32240 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:44319
I20260809 06:45:59.312667 32349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:44319 every 8 connection(s)
I20260809 06:45:59.313975 32240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:45:59.317750 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:45:59.318856 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 32240
I20260809 06:45:59.319069 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:45:59.319758 28487 external_mini_cluster.cc:1428] Setting key f2e0676d46d38464cdf47f468b63b464
I20260809 06:45:59.332151 32350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20: Bootstrap starting.
Aug 09 06:45:59 dist-test-slave-jpj1 krb5kdc[32218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:59.335340 32350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20: Neither blocks nor log segments found. Creating new log.
I20260809 06:45:59.336777 32350 log.cc:824] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20: Log is configured to *not* fsync() on all Append() calls
I20260809 06:45:59.340178 32350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20: No bootstrap required, opened a new log
I20260809 06:45:59.343554 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:59.321840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.27.209.254:44319)
0809 06:45:59.322678 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:59.322692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:59.322748 (+ 56us) server_negotiation.cc:407] Connection header received
0809 06:45:59.324452 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:59.324491 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:59.325011 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:59.325492 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:59.326839 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:59.328897 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:59.330195 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:59.331240 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:59.331419 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:59.335258 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:59.335270 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:59.335291 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:59.335343 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:59.338977 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:59.339688 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:59.339697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:59.339706 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:59.339927 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:59.340278 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:59.340283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:59.340286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:59.341010 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:59.341325 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:59.341708 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 06:45:59.342175 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":352,"threads_started":1}
I20260809 06:45:59.349030 32350 raft_consensus.cc:348] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793dac6be174f8c8f3ccee465a99e20" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44319 } }
I20260809 06:45:59.349372 32350 raft_consensus.cc:374] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:45:59.349448 32350 raft_consensus.cc:729] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a793dac6be174f8c8f3ccee465a99e20, State: Initialized, Role: FOLLOWER
I20260809 06:45:59.349977 32350 consensus_queue.cc:260] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793dac6be174f8c8f3ccee465a99e20" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44319 } }
I20260809 06:45:59.350183 32350 raft_consensus.cc:388] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:45:59.350275 32350 raft_consensus.cc:482] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:45:59.350400 32350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:45:59.352802 32350 raft_consensus.cc:504] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793dac6be174f8c8f3ccee465a99e20" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44319 } }
I20260809 06:45:59.353251 32350 leader_election.cc:302] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a793dac6be174f8c8f3ccee465a99e20; no voters:
I20260809 06:45:59.353783 32350 leader_election.cc:288] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:45:59.353907 32358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:45:59.354508 32358 raft_consensus.cc:686] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [term 1 LEADER]: Becoming Leader. State: Replica: a793dac6be174f8c8f3ccee465a99e20, State: Running, Role: LEADER
I20260809 06:45:59.355042 32358 consensus_queue.cc:237] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793dac6be174f8c8f3ccee465a99e20" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44319 } }
I20260809 06:45:59.355793 32350 sys_catalog.cc:564] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:45:59.358129 32359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a793dac6be174f8c8f3ccee465a99e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793dac6be174f8c8f3ccee465a99e20" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44319 } } }
I20260809 06:45:59.358529 32359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:59.358999 32360 sys_catalog.cc:455] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a793dac6be174f8c8f3ccee465a99e20. Latest consensus state: current_term: 1 leader_uuid: "a793dac6be174f8c8f3ccee465a99e20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a793dac6be174f8c8f3ccee465a99e20" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44319 } } }
I20260809 06:45:59.359385 32360 sys_catalog.cc:458] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20 [sys.catalog]: This master's current role is: LEADER
I20260809 06:45:59.360908 32369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:45:59.365633 32369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:45:59.373713 32369 catalog_manager.cc:1347] Generated new cluster ID: 5250de15f177440980ab980f2036ab3a
I20260809 06:45:59.373876 32369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:45:59.395131 32369 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:45:59.396525 32369 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:45:59.410902 32369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a793dac6be174f8c8f3ccee465a99e20: Generated new TSK 0
I20260809 06:45:59.411751 32369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:45:59.458671 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44319
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:45:59.785029 32387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:45:59.785288 32387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:45:59.785386 32387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:45:59.792430 32387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:45:59.792536 32387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:45:59.792656 32387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:45:59.802103 32387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:45:59.803275 32387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:45:59.804593 32387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:45:59.817226 32396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:59.817204 32398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:45:59.817514 32394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:45:59.817924 32387 server_base.cc:1034] running on GCE node
I20260809 06:45:59.818523 32387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:45:59.819461 32387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:45:59.820694 32387 hybrid_clock.cc:648] HybridClock initialized: now 1786257959820618 us; error 39 us; skew 500 ppm
I20260809 06:45:59.820978 32387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:45:59 dist-test-slave-jpj1 krb5kdc[32218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:45:59.824358 32387 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:45:59.835917 32387 webserver.cc:460] Webserver started at http://127.27.209.193:40873/ using document root <none> and password file <none>
I20260809 06:45:59.836956 32387 fs_manager.cc:359] Metadata directory not provided
I20260809 06:45:59.837146 32387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:45:59.837559 32387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:45:59.841697 32387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "91abb714ac124f7897141ad18977d7a1"
format_stamp: "Formatted at 2026-08-09 06:45:59 on dist-test-slave-jpj1"
server_key: "1684b3af9764265657476a0221694dd5"
server_key_iv: "cf056b638cd0f947f3cf8e04eaee5089"
server_key_version: "encryptionkey@0"
I20260809 06:45:59.842702 32387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "91abb714ac124f7897141ad18977d7a1"
format_stamp: "Formatted at 2026-08-09 06:45:59 on dist-test-slave-jpj1"
server_key: "1684b3af9764265657476a0221694dd5"
server_key_iv: "cf056b638cd0f947f3cf8e04eaee5089"
server_key_version: "encryptionkey@0"
I20260809 06:45:59.849440 32387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 06:45:59.853888 32406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:59.855460 32387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:45:59.855650 32387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "91abb714ac124f7897141ad18977d7a1"
format_stamp: "Formatted at 2026-08-09 06:45:59 on dist-test-slave-jpj1"
server_key: "1684b3af9764265657476a0221694dd5"
server_key_iv: "cf056b638cd0f947f3cf8e04eaee5089"
server_key_version: "encryptionkey@0"
I20260809 06:45:59.855878 32387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:45:59.881289 32387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:45:59.886376 32387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:45:59.886667 32387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:45:59.887827 32387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:45:59.889513 32387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:45:59.889621 32387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:59.889755 32387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:45:59.889822 32387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:45:59.921492 32387 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:36673
I20260809 06:45:59.921506 32575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:36673 every 8 connection(s)
I20260809 06:45:59.923112 32387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:45:59.928622 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 32387
I20260809 06:45:59.928825 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:45:59.929549 28487 external_mini_cluster.cc:1428] Setting key 3cae9985bd4e0c7c7d6d40280b4367ff
Aug 09 06:45:59 dist-test-slave-jpj1 krb5kdc[32218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:45:59.951713 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:59.935722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:41327 (local address 127.27.209.254:44319)
0809 06:45:59.936157 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0809 06:45:59.936163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:45:59.936185 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:45:59.936923 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:45:59.936929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:45:59.937023 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:45:59.937132 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:45:59.939193 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:59.940036 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:59.942321 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:45:59.942727 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:45:59.942856 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:59.946227 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:45:59.946233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:45:59.946236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:45:59.946268 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:45:59.948700 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:59.949393 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:59.949397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:59.949400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:59.949518 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:45:59.949951 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:45:59.949956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:45:59.949958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:45:59.950215 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:45:59.950373 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:45:59.951106 (+ 733us) server_negotiation.cc:299] Negotiation successful
0809 06:45:59.951341 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":201,"threads_started":1}
I20260809 06:45:59.952708 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:45:59.933415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44319 (local address 127.27.209.193:41327)
0809 06:45:59.934264 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0809 06:45:59.934325 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 06:45:59.936595 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:45:59.937551 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:45:59.937563 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:45:59.938261 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:45:59.939013 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:59.939035 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:59.940219 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:59.940225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:59.942165 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:45:59.942182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:45:59.942960 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:45:59.942968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:45:59.943202 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:45:59.943874 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:45:59.943902 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:45:59.946015 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:45:59.948842 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:59.948851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:59.948867 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:59.949271 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:59.949634 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:45:59.949639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:45:59.949643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:45:59.949843 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:45:59.950344 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:45:59.950352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:45:59.950683 (+ 331us) client_negotiation.cc:769] Sending connection context
0809 06:45:59.950996 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 06:45:59.951505 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":1611,"threads_started":1}
I20260809 06:45:59.954242 32576 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44319
I20260809 06:45:59.954545 32576 heartbeater.cc:464] Registering TS with master...
I20260809 06:45:59.955255 32576 heartbeater.cc:511] Master 127.27.209.254:44319 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
I20260809 06:45:59.957581 32296 ts_manager.cc:194] Registered new tserver with Master: 91abb714ac124f7897141ad18977d7a1 (127.27.209.193:36673)
I20260809 06:45:59.959048 32296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:41327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:45:59.982393 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44319
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:00.304649 32583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:00.304930 32583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:00.305229 32583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:00.312885 32583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:00.313030 32583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:00.313230 32583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:00.328246 32583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:00.329870 32583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:00.331516 32583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:00.344733 32590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:00.344765 32592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:00.344931 32596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:00.345655 32583 server_base.cc:1034] running on GCE node
I20260809 06:46:00.346274 32583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:00.347234 32583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:00.348435 32583 hybrid_clock.cc:648] HybridClock initialized: now 1786257960348390 us; error 31 us; skew 500 ppm
I20260809 06:46:00.348757 32583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:00 dist-test-slave-jpj1 krb5kdc[32218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:00.351857 32583 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:46:00.353303 32583 webserver.cc:460] Webserver started at http://127.27.209.194:46633/ using document root <none> and password file <none>
I20260809 06:46:00.353992 32583 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:00.354112 32583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:00.354420 32583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:00.357506 32583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "8b43f03e443a4187beab7fc85a2ab3ca"
format_stamp: "Formatted at 2026-08-09 06:46:00 on dist-test-slave-jpj1"
server_key: "e29c671944bd8c4f96460651e0b92311"
server_key_iv: "0f7dfcfff36289ec2c3a7720db590d62"
server_key_version: "encryptionkey@0"
I20260809 06:46:00.358278 32583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "8b43f03e443a4187beab7fc85a2ab3ca"
format_stamp: "Formatted at 2026-08-09 06:46:00 on dist-test-slave-jpj1"
server_key: "e29c671944bd8c4f96460651e0b92311"
server_key_iv: "0f7dfcfff36289ec2c3a7720db590d62"
server_key_version: "encryptionkey@0"
I20260809 06:46:00.363648 32583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:46:00.367065 32604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:00.368337 32583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:00.368487 32583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "8b43f03e443a4187beab7fc85a2ab3ca"
format_stamp: "Formatted at 2026-08-09 06:46:00 on dist-test-slave-jpj1"
server_key: "e29c671944bd8c4f96460651e0b92311"
server_key_iv: "0f7dfcfff36289ec2c3a7720db590d62"
server_key_version: "encryptionkey@0"
I20260809 06:46:00.368662 32583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:00.389258 32583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:00.393569 32583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:00.393846 32583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:00.394815 32583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:00.396173 32583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:00.396257 32583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:00.396394 32583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:00.396447 32583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:00.425294 32583 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:42681
I20260809 06:46:00.425338 32765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:42681 every 8 connection(s)
I20260809 06:46:00.426532 32583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:00.429211 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 32583
I20260809 06:46:00.429406 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:00.429975 28487 external_mini_cluster.cc:1428] Setting key c8b64d336e97a665bc6c2c7bca93093b
Aug 09 06:46:00 dist-test-slave-jpj1 krb5kdc[32218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:00.446484 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:00.429029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:53461 (local address 127.27.209.254:44319)
0809 06:46:00.429189 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:00.429194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:00.430692 (+ 1498us) server_negotiation.cc:407] Connection header received
0809 06:46:00.432739 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:00.432746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:00.432836 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:00.432923 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:00.434434 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.435140 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.436728 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.437163 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.437276 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:00.440964 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:00.440969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:00.440972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:00.441006 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:00.443487 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:00.444209 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:00.444213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:00.444215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:00.444347 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:00.444763 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:00.444767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:00.444769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:00.445056 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:00.445192 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:00.445906 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 06:46:00.446140 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:46:00.447422 300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:00.429710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44319 (local address 127.27.209.194:53461)
0809 06:46:00.430503 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:00.430545 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:00.432468 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:00.433172 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:00.433180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:00.433696 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:00.434247 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:00.434265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.435286 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.435290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:00.436583 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:00.436596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.437422 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.437433 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:00.437646 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:00.438296 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:00.438336 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:00.440744 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:00.443636 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:00.443645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:00.443661 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:00.444083 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:00.444455 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:00.444460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:00.444463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:00.444646 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:00.445185 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:00.445193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:00.445515 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 06:46:00.445756 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 06:46:00.446269 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":321,"threads_started":1}
I20260809 06:46:00.448854 32766 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44319
I20260809 06:46:00.449150 32766 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
I20260809 06:46:00.449894 32766 heartbeater.cc:511] Master 127.27.209.254:44319 requested a full tablet report, sending...
I20260809 06:46:00.451296 32296 ts_manager.cc:194] Registered new tserver with Master: 8b43f03e443a4187beab7fc85a2ab3ca (127.27.209.194:42681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
I20260809 06:46:00.452368 32296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:53461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:46:00.474452 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44319
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:00.791366 308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:00.791688 308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:00.791801 308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:00.798723 308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:00.798825 308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:00.798940 308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:00.808243 308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:00.809409 308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:00.810750 308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:00.823060 314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:00.822986 316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:00.823004 321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:00.823989 308 server_base.cc:1034] running on GCE node
I20260809 06:46:00.824528 308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:00.825364 308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:00.826555 308 hybrid_clock.cc:648] HybridClock initialized: now 1786257960826507 us; error 37 us; skew 500 ppm
I20260809 06:46:00.826826 308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:00 dist-test-slave-jpj1 krb5kdc[32218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:00.829751 308 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:46:00.831073 308 webserver.cc:460] Webserver started at http://127.27.209.195:34275/ using document root <none> and password file <none>
I20260809 06:46:00.831754 308 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:00.831856 308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:00.832125 308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:00.834877 308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "a16be91c0ab44d7fa02347d591f4764d"
format_stamp: "Formatted at 2026-08-09 06:46:00 on dist-test-slave-jpj1"
server_key: "62ed3f4ed19f417656a5d2ed86a6c5a1"
server_key_iv: "67b25e63f1a1253128bbfcd0c16ddc1a"
server_key_version: "encryptionkey@0"
I20260809 06:46:00.835639 308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "a16be91c0ab44d7fa02347d591f4764d"
format_stamp: "Formatted at 2026-08-09 06:46:00 on dist-test-slave-jpj1"
server_key: "62ed3f4ed19f417656a5d2ed86a6c5a1"
server_key_iv: "67b25e63f1a1253128bbfcd0c16ddc1a"
server_key_version: "encryptionkey@0"
I20260809 06:46:00.840540 308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:46:00.843858 334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:00.845077 308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:00.845228 308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "a16be91c0ab44d7fa02347d591f4764d"
format_stamp: "Formatted at 2026-08-09 06:46:00 on dist-test-slave-jpj1"
server_key: "62ed3f4ed19f417656a5d2ed86a6c5a1"
server_key_iv: "67b25e63f1a1253128bbfcd0c16ddc1a"
server_key_version: "encryptionkey@0"
I20260809 06:46:00.845415 308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:00.860836 308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:00.864874 308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:00.865127 308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:00.866135 308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:00.867477 308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:00.867579 308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 06:46:00.867708 308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:00.867775 308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:00.897945 308 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:34745
I20260809 06:46:00.897992 490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:34745 every 8 connection(s)
I20260809 06:46:00.899196 308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:00.901852 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 308
I20260809 06:46:00.902047 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:00.902465 28487 external_mini_cluster.cc:1428] Setting key 48c71564fbb56b5c7c8ff8c7ac8cef8b
Aug 09 06:46:00 dist-test-slave-jpj1 krb5kdc[32218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:00.918957 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:00.901782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:42057 (local address 127.27.209.254:44319)
0809 06:46:00.902013 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:00.902019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:00.903258 (+ 1239us) server_negotiation.cc:407] Connection header received
0809 06:46:00.905481 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:00.905486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:00.905570 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:00.905655 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:00.907152 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.907938 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.909442 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.909854 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.909965 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:00.913250 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:00.913256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:00.913259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:00.913291 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:00.916090 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:00.916756 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:00.916760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:00.916763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:00.916891 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:00.917295 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:00.917299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:00.917301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:00.917588 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:00.917723 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:00.918405 (+ 682us) server_negotiation.cc:299] Negotiation successful
0809 06:46:00.918628 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:46:00.919888 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:00.902278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44319 (local address 127.27.209.195:42057)
0809 06:46:00.903096 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:00.903145 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:00.905196 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:00.905917 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:00.905925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:00.906429 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:00.906991 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:00.907009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.908094 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.908098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:00.909300 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:00.909313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:00.910071 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:00.910078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:00.910304 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:00.910925 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:00.910948 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:00.913054 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:00.916221 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:00.916230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:00.916245 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:00.916640 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:00.916997 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:00.917002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:00.917005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:00.917185 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:00.917722 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:00.917730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:00.918043 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 06:46:00.918284 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 06:46:00.918768 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":297,"threads_started":1}
I20260809 06:46:00.921312 494 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44319
I20260809 06:46:00.921626 494 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:00.922339 494 heartbeater.cc:511] Master 127.27.209.254:44319 requested a full tablet report, sending...
I20260809 06:46:00.923806 32296 ts_manager.cc:194] Registered new tserver with Master: a16be91c0ab44d7fa02347d591f4764d (127.27.209.195:34745)
I20260809 06:46:00.924572 32296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:42057
I20260809 06:46:00.929468 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:00.962662 32576 heartbeater.cc:503] Master 127.27.209.254:44319 was elected leader, sending a full tablet report...
W20260809 06:46:00.962965 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:00.960389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.27.209.254:44319)
0809 06:46:00.960607 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:00.960612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:00.961091 (+ 479us) server_negotiation.cc:407] Connection header received
0809 06:46:00.961257 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:00.961261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:00.961351 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:00.961439 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:00.962714 (+ 1275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33246: BlockingRecv error: recv got EOF from 127.0.0.1:33246 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 06:46:00.966406 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 32387
W20260809 06:46:00.969151 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 32387: Not authorized: Client connection negotiation failed: client connection to 127.27.209.193:36673: server requires authentication, but client does not have Kerberos credentials available
I20260809 06:46:00.969327 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 32387
I20260809 06:46:00.991123 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 32583
W20260809 06:46:00.993659 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 32583: Not authorized: Client connection negotiation failed: client connection to 127.27.209.194:42681: server requires authentication, but client does not have Kerberos credentials available
I20260809 06:46:00.993801 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 32583
I20260809 06:46:01.017033 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 308
W20260809 06:46:01.019976 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 308: Not authorized: Client connection negotiation failed: client connection to 127.27.209.195:34745: server requires authentication, but client does not have Kerberos credentials available
I20260809 06:46:01.020103 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 308
I20260809 06:46:01.041574 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 32240
I20260809 06:46:01.042824 32343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:01.149674 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 32240
2026-08-09T06:46:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3635 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:01 dist-test-slave-jpj1 krb5kdc[526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:01 dist-test-slave-jpj1 krb5kdc[526](info): set up 2 sockets
Aug 09 06:46:01 dist-test-slave-jpj1 krb5kdc[526](info): commencing operation
krb5kdc: starting...
W20260809 06:46:02.332671 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/test-user.keytab.
W20260809 06:46:03.122473 28487 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.588s user 0.000s sys 0.006s
Aug 09 06:46:03 dist-test-slave-jpj1 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
W20260809 06:46:04.269177 28487 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.209.254: real 1.090s user 0.000s sys 0.011s
I20260809 06:46:04.270267 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40263
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:34743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:04.590309 547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:04.590602 547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:04.590690 547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:04.597836 547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:04.597936 547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:04.597988 547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:04.598047 547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:04.598094 547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:04.607813 547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:04.609076 547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:04.610389 547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:04.622172 554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:04.622135 555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:04.622208 557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:04.623029 547 server_base.cc:1034] running on GCE node
I20260809 06:46:04.623872 547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:04.625298 547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:04.626497 547 hybrid_clock.cc:648] HybridClock initialized: now 1786257964626450 us; error 39 us; skew 500 ppm
I20260809 06:46:04.626856 547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:04 dist-test-slave-jpj1 krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257964, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:04.633396 547 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:04.634812 547 webserver.cc:460] Webserver started at http://127.27.209.254:39079/ using document root <none> and password file <none>
I20260809 06:46:04.635459 547 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:04.635587 547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:04.635900 547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:04.638820 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "4aa9c69d739840d6aee6c6f7704321a1"
format_stamp: "Formatted at 2026-08-09 06:46:04 on dist-test-slave-jpj1"
server_key: "e3e013d1bdeaa84f05bfdc4bd619b236"
server_key_iv: "ac15ec6bbc52b86508bc7ad8e5382635"
server_key_version: "encryptionkey@0"
I20260809 06:46:04.639573 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "4aa9c69d739840d6aee6c6f7704321a1"
format_stamp: "Formatted at 2026-08-09 06:46:04 on dist-test-slave-jpj1"
server_key: "e3e013d1bdeaa84f05bfdc4bd619b236"
server_key_iv: "ac15ec6bbc52b86508bc7ad8e5382635"
server_key_version: "encryptionkey@0"
I20260809 06:46:04.644569 547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260809 06:46:04.647887 565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:04.649158 547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:04.649319 547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "4aa9c69d739840d6aee6c6f7704321a1"
format_stamp: "Formatted at 2026-08-09 06:46:04 on dist-test-slave-jpj1"
server_key: "e3e013d1bdeaa84f05bfdc4bd619b236"
server_key_iv: "ac15ec6bbc52b86508bc7ad8e5382635"
server_key_version: "encryptionkey@0"
I20260809 06:46:04.649492 547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:04.683874 547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:04.687837 547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:04.688110 547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:04.705010 547 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:40263
I20260809 06:46:04.705053 631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:40263 every 8 connection(s)
I20260809 06:46:04.706359 547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:04.709223 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 547
I20260809 06:46:04.709476 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:04.710345 28487 external_mini_cluster.cc:1428] Setting key c9ca39fb97c082652f95f661fc33981c
I20260809 06:46:04.710704 632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:46:04 dist-test-slave-jpj1 krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:04.725310 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1: Bootstrap starting.
I20260809 06:46:04.728816 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:04.730018 632 log.cc:824] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:04.733269 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1: No bootstrap required, opened a new log
I20260809 06:46:04.733687 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:04.712216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.27.209.254:40263)
0809 06:46:04.712978 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:04.712990 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:04.713030 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:46:04.715251 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:04.715278 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:04.715705 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:04.716054 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:04.717980 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:04.719250 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:04.721002 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:04.721695 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:04.721852 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:04.726517 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:04.726525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:04.726540 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:04.726582 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:04.729496 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:04.730314 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:04.730321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:04.730328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:04.730489 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:04.730931 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:04.730936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:04.730938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:04.731606 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:04.731854 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:04.732278 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 06:46:04.732619 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":326,"threads_started":1}
I20260809 06:46:04.737445 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:46:04.742611 632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa9c69d739840d6aee6c6f7704321a1" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40263 } }
I20260809 06:46:04.743100 632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:04.743203 632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa9c69d739840d6aee6c6f7704321a1, State: Initialized, Role: FOLLOWER
I20260809 06:46:04.743901 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa9c69d739840d6aee6c6f7704321a1" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40263 } }
I20260809 06:46:04.744174 632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:04.744302 632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:04.744473 632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:04.747282 632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa9c69d739840d6aee6c6f7704321a1" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40263 } }
I20260809 06:46:04.747810 632 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa9c69d739840d6aee6c6f7704321a1; no voters:
I20260809 06:46:04.748428 632 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:04.748536 642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:04.750890 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:04.755467 642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [term 1 LEADER]: Becoming Leader. State: Replica: 4aa9c69d739840d6aee6c6f7704321a1, State: Running, Role: LEADER
I20260809 06:46:04.756242 642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa9c69d739840d6aee6c6f7704321a1" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40263 } }
I20260809 06:46:04.759456 643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa9c69d739840d6aee6c6f7704321a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa9c69d739840d6aee6c6f7704321a1" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40263 } } }
I20260809 06:46:04.759919 643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:04.760406 645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa9c69d739840d6aee6c6f7704321a1. Latest consensus state: current_term: 1 leader_uuid: "4aa9c69d739840d6aee6c6f7704321a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa9c69d739840d6aee6c6f7704321a1" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40263 } } }
I20260809 06:46:04.760679 645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:04.764855 665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:04.768996 665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:04.776954 665 catalog_manager.cc:1347] Generated new cluster ID: 09ad4590d823421d91a7b44007a59944
I20260809 06:46:04.777076 665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:04.786877 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:04.779807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.27.209.254:40263)
0809 06:46:04.780019 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:04.780025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:04.780593 (+ 568us) server_negotiation.cc:407] Connection header received
0809 06:46:04.780769 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:04.780773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:04.780841 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:04.780928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:04.781946 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:04.782656 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:04.783961 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:04.784401 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:04.784510 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:04.785499 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:04.785504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:04.785507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:04.786131 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:04.786180 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:04.786207 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:04.786310 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:04.786454 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 06:46:04.786551 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:46:04.790422 665 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:04.791747 665 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:04.803736 665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa9c69d739840d6aee6c6f7704321a1: Generated new TSK 0
I20260809 06:46:04.804603 665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:46:04.814607 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 547
I20260809 06:46:04.815899 626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:04.903863 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 547
2026-08-09T06:46:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3747 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:04 dist-test-slave-jpj1 krb5kdc[686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:04 dist-test-slave-jpj1 krb5kdc[686](info): set up 2 sockets
Aug 09 06:46:04 dist-test-slave-jpj1 krb5kdc[686](info): commencing operation
krb5kdc: starting...
W20260809 06:46:06.074446 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:06 dist-test-slave-jpj1 krb5kdc[686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:46:06.270974 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:43657
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:06.605083 703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:06.605361 703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:06.605491 703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:06.612483 703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:06.612589 703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:06.612668 703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:06.612740 703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:06.612805 703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:06.622195 703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:06.623443 703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:06.624751 703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:06.637742 714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:06.637667 710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:06.637779 708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:06.639240 703 server_base.cc:1034] running on GCE node
I20260809 06:46:06.640035 703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:06.641335 703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:06.642529 703 hybrid_clock.cc:648] HybridClock initialized: now 1786257966642485 us; error 36 us; skew 500 ppm
I20260809 06:46:06.642860 703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:06 dist-test-slave-jpj1 krb5kdc[686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:06.645864 703 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:06.647297 703 webserver.cc:460] Webserver started at http://127.27.209.254:36263/ using document root <none> and password file <none>
I20260809 06:46:06.647997 703 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:06.648114 703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:06.648397 703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:06.651311 703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "83399f5f31664704a2a3ed734837f4cf"
format_stamp: "Formatted at 2026-08-09 06:46:06 on dist-test-slave-jpj1"
server_key: "509cdff4e47afff1ca19dccdaaf1f402"
server_key_iv: "aab78c22d5f032923692ad17627d51d6"
server_key_version: "encryptionkey@0"
I20260809 06:46:06.652078 703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "83399f5f31664704a2a3ed734837f4cf"
format_stamp: "Formatted at 2026-08-09 06:46:06 on dist-test-slave-jpj1"
server_key: "509cdff4e47afff1ca19dccdaaf1f402"
server_key_iv: "aab78c22d5f032923692ad17627d51d6"
server_key_version: "encryptionkey@0"
I20260809 06:46:06.657079 703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 06:46:06.660518 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:06.661809 703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:46:06.661970 703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "83399f5f31664704a2a3ed734837f4cf"
format_stamp: "Formatted at 2026-08-09 06:46:06 on dist-test-slave-jpj1"
server_key: "509cdff4e47afff1ca19dccdaaf1f402"
server_key_iv: "aab78c22d5f032923692ad17627d51d6"
server_key_version: "encryptionkey@0"
I20260809 06:46:06.662146 703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:06.693253 703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:06.697432 703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:06.697707 703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:06.714820 703 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:43657
I20260809 06:46:06.714855 795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:43657 every 8 connection(s)
I20260809 06:46:06.716151 703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:06.718674 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 703
I20260809 06:46:06.718859 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:06.719491 28487 external_mini_cluster.cc:1428] Setting key 7ab6f5dece50d5dbe033f6e780dbde28
I20260809 06:46:06.719950 797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:06.734025 797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf: Bootstrap starting.
Aug 09 06:46:06 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:06.737507 797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:06.738575 797 log.cc:824] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:06.741549 797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf: No bootstrap required, opened a new log
I20260809 06:46:06.742264 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:06.721365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.27.209.254:43657)
0809 06:46:06.722097 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:06.722109 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:06.722148 (+ 39us) server_negotiation.cc:407] Connection header received
0809 06:46:06.724379 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:06.724402 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:06.724772 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:06.725104 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:06.726869 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:06.728158 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:06.729868 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:06.730540 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:06.730667 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:06.735190 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:06.735198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:06.735213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:06.735248 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:06.738291 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:06.738969 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:06.738975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:06.738982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:06.739151 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:06.739529 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:06.739560 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:06.739564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:06.740110 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:06.740391 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:06.740850 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 06:46:06.741227 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":316,"threads_started":1}
I20260809 06:46:06.749384 797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83399f5f31664704a2a3ed734837f4cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43657 } }
I20260809 06:46:06.749754 797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:06.749855 797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83399f5f31664704a2a3ed734837f4cf, State: Initialized, Role: FOLLOWER
I20260809 06:46:06.750402 797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83399f5f31664704a2a3ed734837f4cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43657 } }
I20260809 06:46:06.750690 797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:06.750826 797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:06.751008 797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:06.753175 797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83399f5f31664704a2a3ed734837f4cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43657 } }
I20260809 06:46:06.753654 797 leader_election.cc:302] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83399f5f31664704a2a3ed734837f4cf; no voters:
I20260809 06:46:06.754189 797 leader_election.cc:288] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:06.754311 805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:06.754932 805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [term 1 LEADER]: Becoming Leader. State: Replica: 83399f5f31664704a2a3ed734837f4cf, State: Running, Role: LEADER
I20260809 06:46:06.755431 805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83399f5f31664704a2a3ed734837f4cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43657 } }
I20260809 06:46:06.756219 797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:06.758338 806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83399f5f31664704a2a3ed734837f4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83399f5f31664704a2a3ed734837f4cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43657 } } }
I20260809 06:46:06.758742 806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:06.758556 807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83399f5f31664704a2a3ed734837f4cf. Latest consensus state: current_term: 1 leader_uuid: "83399f5f31664704a2a3ed734837f4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83399f5f31664704a2a3ed734837f4cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43657 } } }
I20260809 06:46:06.758868 807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:06.760427 810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:06.765501 810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:06.774317 810 catalog_manager.cc:1347] Generated new cluster ID: fa782412e7384addb5a5385f42591039
I20260809 06:46:06.774420 810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:06.787176 810 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:06.788502 810 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:06.807010 810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83399f5f31664704a2a3ed734837f4cf: Generated new TSK 0
I20260809 06:46:06.807919 810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:46:06.878234 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43657
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:07.214776 837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:07.215061 837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:07.215214 837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:07.222674 837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:07.222805 837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:07.222970 837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:07.232841 837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:07.234138 837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:07.235445 837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:07.248188 850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:07.248263 845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:07.248235 846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:07.248998 837 server_base.cc:1034] running on GCE node
I20260809 06:46:07.249615 837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:07.250526 837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:07.251727 837 hybrid_clock.cc:648] HybridClock initialized: now 1786257967251668 us; error 56 us; skew 500 ppm
I20260809 06:46:07.252027 837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:07 dist-test-slave-jpj1 krb5kdc[686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:07.255201 837 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:46:07.256727 837 webserver.cc:460] Webserver started at http://127.27.209.193:34657/ using document root <none> and password file <none>
I20260809 06:46:07.257468 837 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:07.257591 837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:07.257895 837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:07.261053 837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "0bfe4323217b47108d7d8e65901de91b"
format_stamp: "Formatted at 2026-08-09 06:46:07 on dist-test-slave-jpj1"
server_key: "da0dfb67a7123b583058a855e2a70354"
server_key_iv: "1e7ffe746fa505459188951c8e7062d4"
server_key_version: "encryptionkey@0"
I20260809 06:46:07.261848 837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "0bfe4323217b47108d7d8e65901de91b"
format_stamp: "Formatted at 2026-08-09 06:46:07 on dist-test-slave-jpj1"
server_key: "da0dfb67a7123b583058a855e2a70354"
server_key_iv: "1e7ffe746fa505459188951c8e7062d4"
server_key_version: "encryptionkey@0"
I20260809 06:46:07.267030 837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:46:07.270408 860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:07.271716 837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:07.271873 837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "0bfe4323217b47108d7d8e65901de91b"
format_stamp: "Formatted at 2026-08-09 06:46:07 on dist-test-slave-jpj1"
server_key: "da0dfb67a7123b583058a855e2a70354"
server_key_iv: "1e7ffe746fa505459188951c8e7062d4"
server_key_version: "encryptionkey@0"
I20260809 06:46:07.272042 837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:07.286247 837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:07.289914 837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:07.290184 837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:07.291097 837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:07.292441 837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:07.292519 837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:07.292649 837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:07.292708 837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:07.322090 837 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:43511
I20260809 06:46:07.322134 1039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:43511 every 8 connection(s)
I20260809 06:46:07.323378 837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:07.331267 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 837
I20260809 06:46:07.331491 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:07.332135 28487 external_mini_cluster.cc:1428] Setting key f027d14d8d3811721a72827fc88d297e
Aug 09 06:46:07 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:07.344259 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:07.325930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:41483 (local address 127.27.209.254:43657)
0809 06:46:07.326426 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:07.326432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:07.327484 (+ 1052us) server_negotiation.cc:407] Connection header received
0809 06:46:07.329349 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:07.329354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:07.329429 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:07.329532 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:07.331260 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.332083 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.333858 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.334367 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.334511 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:07.338058 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:07.338065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:07.338069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:07.338111 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:07.341078 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:07.341886 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:07.341890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:07.341892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:07.342027 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:07.342515 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:07.342518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:07.342520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:07.342778 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:07.342907 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:07.343687 (+ 780us) server_negotiation.cc:299] Negotiation successful
0809 06:46:07.343916 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260809 06:46:07.345077 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:07.326526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:43657 (local address 127.27.209.193:41483)
0809 06:46:07.327299 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:07.327347 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:07.329058 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:07.329761 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:07.329770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:07.330338 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:07.331078 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:07.331107 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.332253 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.332259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:07.333692 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:07.333706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.334593 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.334599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:07.334813 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:07.335450 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:07.335473 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:07.337816 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:07.341214 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:07.341220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:07.341234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:07.341770 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:07.342125 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:07.342129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:07.342133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:07.342416 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:07.342899 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:07.342905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:07.343243 (+ 338us) client_negotiation.cc:769] Sending connection context
0809 06:46:07.343518 (+ 275us) client_negotiation.cc:240] Negotiation successful
0809 06:46:07.343958 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":308,"threads_started":1}
I20260809 06:46:07.346673 1042 heartbeater.cc:347] Connected to a master server at 127.27.209.254:43657
I20260809 06:46:07.347009 1042 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:07.347932 1042 heartbeater.cc:511] Master 127.27.209.254:43657 requested a full tablet report, sending...
I20260809 06:46:07.350330 753 ts_manager.cc:194] Registered new tserver with Master: 0bfe4323217b47108d7d8e65901de91b (127.27.209.193:43511)
I20260809 06:46:07.351801 753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:41483
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:46:07.378533 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43657
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:07.701812 1049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:07.702080 1049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:07.702183 1049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:07.709160 1049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:07.709257 1049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:07.709371 1049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:07.718549 1049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:07.719769 1049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:07.720993 1049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:07.733174 1062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:07.733203 1055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:07.733371 1054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:07.733882 1049 server_base.cc:1034] running on GCE node
I20260809 06:46:07.734414 1049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:07.735219 1049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:07.736397 1049 hybrid_clock.cc:648] HybridClock initialized: now 1786257967736342 us; error 51 us; skew 500 ppm
I20260809 06:46:07.736675 1049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:07 dist-test-slave-jpj1 krb5kdc[686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:07.739651 1049 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:46:07.740958 1049 webserver.cc:460] Webserver started at http://127.27.209.194:41561/ using document root <none> and password file <none>
I20260809 06:46:07.741596 1049 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:07.741716 1049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:07.742004 1049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:07.744843 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "57f31e6255d64695a92284e124e197dc"
format_stamp: "Formatted at 2026-08-09 06:46:07 on dist-test-slave-jpj1"
server_key: "59e2f8c0ed32912875980a92a183a00c"
server_key_iv: "7c359aac1502ecdc76d1aff7e0b7685b"
server_key_version: "encryptionkey@0"
I20260809 06:46:07.745580 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "57f31e6255d64695a92284e124e197dc"
format_stamp: "Formatted at 2026-08-09 06:46:07 on dist-test-slave-jpj1"
server_key: "59e2f8c0ed32912875980a92a183a00c"
server_key_iv: "7c359aac1502ecdc76d1aff7e0b7685b"
server_key_version: "encryptionkey@0"
I20260809 06:46:07.750701 1049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:46:07.753996 1070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:07.755234 1049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 06:46:07.755383 1049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "57f31e6255d64695a92284e124e197dc"
format_stamp: "Formatted at 2026-08-09 06:46:07 on dist-test-slave-jpj1"
server_key: "59e2f8c0ed32912875980a92a183a00c"
server_key_iv: "7c359aac1502ecdc76d1aff7e0b7685b"
server_key_version: "encryptionkey@0"
I20260809 06:46:07.755609 1049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:07.773167 1049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:07.777137 1049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:07.777427 1049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:07.778322 1049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:07.779636 1049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:07.779717 1049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:07.779839 1049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:07.779893 1049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:07.810015 1049 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:38579
I20260809 06:46:07.810051 1244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:38579 every 8 connection(s)
I20260809 06:46:07.811287 1049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:07.815276 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 1049
I20260809 06:46:07.815428 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:07.815877 28487 external_mini_cluster.cc:1428] Setting key 73c8d2eac718bb025fb220b88ba98a26
Aug 09 06:46:07 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:07.830823 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:07.813637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:41191 (local address 127.27.209.254:43657)
0809 06:46:07.813854 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:07.813860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:07.815007 (+ 1147us) server_negotiation.cc:407] Connection header received
0809 06:46:07.816867 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:07.816871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:07.816939 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:07.817018 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:07.818592 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.819289 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.820840 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.821246 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.821357 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:07.824957 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:07.824963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:07.824966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:07.824998 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:07.827879 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:07.828564 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:07.828569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:07.828573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:07.828720 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:07.829106 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:07.829110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:07.829113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:07.829451 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:07.829608 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:07.830245 (+ 637us) server_negotiation.cc:299] Negotiation successful
0809 06:46:07.830476 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:46:07.831614 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:07.814153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:43657 (local address 127.27.209.194:41191)
0809 06:46:07.814843 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:07.814882 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:07.816634 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:07.817275 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:07.817286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:07.817881 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:07.818430 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:07.818448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.819437 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.819441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:07.820695 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:07.820708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:07.821466 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:07.821474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:07.821693 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:07.822307 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:07.822333 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:07.824669 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:07.828005 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:07.828011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:07.828024 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:07.828437 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:07.828814 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:07.828817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:07.828819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:07.828988 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:07.829556 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:07.829561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:07.829897 (+ 336us) client_negotiation.cc:769] Sending connection context
0809 06:46:07.830111 (+ 214us) client_negotiation.cc:240] Negotiation successful
0809 06:46:07.830487 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":258,"threads_started":1}
I20260809 06:46:07.833055 1245 heartbeater.cc:347] Connected to a master server at 127.27.209.254:43657
I20260809 06:46:07.833339 1245 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:07.834052 1245 heartbeater.cc:511] Master 127.27.209.254:43657 requested a full tablet report, sending...
I20260809 06:46:07.835484 753 ts_manager.cc:194] Registered new tserver with Master: 57f31e6255d64695a92284e124e197dc (127.27.209.194:38579)
I20260809 06:46:07.836290 753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:41191
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:46:07.861948 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43657
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:40001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:08.185330 1252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:08.185633 1252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:08.185773 1252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:08.193209 1252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:08.193325 1252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:08.193485 1252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:08.203329 1252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:40001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:08.204582 1252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:08.205875 1252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:08.218575 1259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:08.218536 1264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:08.218631 1260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:08.219601 1252 server_base.cc:1034] running on GCE node
I20260809 06:46:08.220217 1252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:08.221117 1252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:08.222325 1252 hybrid_clock.cc:648] HybridClock initialized: now 1786257968222268 us; error 48 us; skew 500 ppm
I20260809 06:46:08.222633 1252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:08.225673 1252 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:46:08.227020 1252 webserver.cc:460] Webserver started at http://127.27.209.195:39167/ using document root <none> and password file <none>
I20260809 06:46:08.227743 1252 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:08.227867 1252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:08.228168 1252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:08.231040 1252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "81f16bce203948f18b4c40872f0cd6d0"
format_stamp: "Formatted at 2026-08-09 06:46:08 on dist-test-slave-jpj1"
server_key: "97f10ecbb6411f1882fb1b27de43b357"
server_key_iv: "4a7612f2f5acf16d90d085b5758743cc"
server_key_version: "encryptionkey@0"
I20260809 06:46:08.231853 1252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "81f16bce203948f18b4c40872f0cd6d0"
format_stamp: "Formatted at 2026-08-09 06:46:08 on dist-test-slave-jpj1"
server_key: "97f10ecbb6411f1882fb1b27de43b357"
server_key_iv: "4a7612f2f5acf16d90d085b5758743cc"
server_key_version: "encryptionkey@0"
I20260809 06:46:08.237026 1252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:46:08.240454 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:08.241695 1252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:08.241868 1252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "81f16bce203948f18b4c40872f0cd6d0"
format_stamp: "Formatted at 2026-08-09 06:46:08 on dist-test-slave-jpj1"
server_key: "97f10ecbb6411f1882fb1b27de43b357"
server_key_iv: "4a7612f2f5acf16d90d085b5758743cc"
server_key_version: "encryptionkey@0"
I20260809 06:46:08.242035 1252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:08.265641 1252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:08.269421 1252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:08.269701 1252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:08.270629 1252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:08.271992 1252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:08.272070 1252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:08.272186 1252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:08.272240 1252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:08.300557 1252 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:43387
I20260809 06:46:08.300618 1431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:43387 every 8 connection(s)
I20260809 06:46:08.301767 1252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:08.310168 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 1252
I20260809 06:46:08.310348 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:08.310778 28487 external_mini_cluster.cc:1428] Setting key bddb24e19c6b3532a8d1310df469997d
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:08.321106 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:08.304175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:56837 (local address 127.27.209.254:43657)
0809 06:46:08.304364 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:08.304369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:08.305515 (+ 1146us) server_negotiation.cc:407] Connection header received
0809 06:46:08.307307 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:08.307312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:08.307386 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:08.307479 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:08.309172 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.310093 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.311808 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.312363 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.312471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:08.316055 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:08.316060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:08.316063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:08.316100 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:08.318471 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.319135 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.319139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.319141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.319284 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.319684 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.319687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.319689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.319949 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:08.320065 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:08.320680 (+ 615us) server_negotiation.cc:299] Negotiation successful
0809 06:46:08.320843 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:46:08.322141 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:08.304639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:43657 (local address 127.27.209.195:56837)
0809 06:46:08.305352 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:08.305397 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:08.307030 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:08.307715 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:08.307723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:08.308284 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:08.308986 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:08.309006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.310244 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.310249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:08.311659 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:08.311675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.312593 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.312598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:08.312883 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:08.313540 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:08.313572 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:08.315864 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:08.318610 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:08.318617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:08.318630 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:08.319044 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:08.319363 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:08.319367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:08.319369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:08.319593 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:08.320047 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:08.320053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:08.320362 (+ 309us) client_negotiation.cc:769] Sending connection context
0809 06:46:08.320585 (+ 223us) client_negotiation.cc:240] Negotiation successful
0809 06:46:08.320969 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":272,"threads_started":1}
I20260809 06:46:08.323508 1432 heartbeater.cc:347] Connected to a master server at 127.27.209.254:43657
I20260809 06:46:08.323835 1432 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:08.324622 1432 heartbeater.cc:511] Master 127.27.209.254:43657 requested a full tablet report, sending...
I20260809 06:46:08.326287 750 ts_manager.cc:194] Registered new tserver with Master: 81f16bce203948f18b4c40872f0cd6d0 (127.27.209.195:43387)
I20260809 06:46:08.327406 750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:56837
I20260809 06:46:08.338550 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:08.355338 1042 heartbeater.cc:503] Master 127.27.209.254:43657 was elected leader, sending a full tablet report...
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:08.389374 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:08.375783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.27.209.254:43657)
0809 06:46:08.375956 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:08.375963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:08.376421 (+ 458us) server_negotiation.cc:407] Connection header received
0809 06:46:08.376604 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:08.376608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:08.376678 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:08.376763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:08.378196 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.378938 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.380222 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.380651 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.380768 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:08.384407 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:08.384413 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:08.384416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:08.384449 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:08.386827 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.387430 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.387435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.387438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.387606 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.387977 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.387981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.387983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.388302 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:08.388541 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:08.388846 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 06:46:08.389055 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 06:46:08.389909 750 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52154
I20260809 06:46:08.391436 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 837
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:08.405949 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:08.392461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.27.209.193:43511)
0809 06:46:08.393116 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:08.393123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:08.393157 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:46:08.393214 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:08.393220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:08.393420 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:08.393543 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:08.394599 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.395642 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.396716 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.397309 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.397406 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:08.401025 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:08.401043 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:08.401062 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:08.401112 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:08.403731 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.404322 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.404329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.404331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.404451 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.404718 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.404721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.404723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.405046 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:08.405279 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:08.405450 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 06:46:08.405657 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":348,"threads_started":1}
W20260809 06:46:08.407287 1033 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38844
W20260809 06:46:08.408387 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:46:08.408597 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 837
I20260809 06:46:08.431206 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1049
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:46:08.446373 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:08.432107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.27.209.194:38579)
0809 06:46:08.432699 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:08.432706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:08.432731 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:46:08.432798 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:08.432805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:08.433004 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:08.433145 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:08.434389 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.435405 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.436672 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.437219 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.437336 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:08.441236 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:08.441247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:08.441261 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:08.441293 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:08.444005 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.444550 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.444557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.444559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.444684 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.444956 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.444958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.444960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.445335 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:08.445621 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:08.445796 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 06:46:08.445984 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":329,"threads_started":1}
W20260809 06:46:08.447666 1237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38906
W20260809 06:46:08.448730 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:46:08.448881 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 1049
I20260809 06:46:08.469853 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1252
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257968, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:46:08.485373 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:08.470777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.27.209.195:43387)
0809 06:46:08.471374 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:08.471382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:08.471413 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:46:08.471476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:08.471483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:08.471742 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:08.471876 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:08.473092 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.474177 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.475354 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:08.476021 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:08.476127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:08.480207 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:08.480219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:08.480234 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:08.480272 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:08.482981 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.483585 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.483591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.483594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.483720 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:08.484029 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:08.484032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:08.484034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:08.484388 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:08.484606 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:08.484809 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 06:46:08.485003 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":296,"threads_started":1}
W20260809 06:46:08.486608 1427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43040
W20260809 06:46:08.487726 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:46:08.487884 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 1252
I20260809 06:46:08.520419 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 703
I20260809 06:46:08.521651 790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:08.623265 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 703
2026-08-09T06:46:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3724 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[1453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[1453](info): set up 2 sockets
Aug 09 06:46:08 dist-test-slave-jpj1 krb5kdc[1453](info): commencing operation
krb5kdc: starting...
W20260809 06:46:09.807345 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:09 dist-test-slave-jpj1 krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:46:10.007257 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:33313
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:10.329532 1470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:10.329873 1470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:10.330003 1470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:10.337272 1470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:10.337389 1470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:10.337446 1470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:10.337527 1470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:10.337589 1470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:10.347388 1470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:10.348726 1470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:10.350064 1470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:10.361500 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:10.361545 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:10.361510 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:10.364781 1470 server_base.cc:1034] running on GCE node
I20260809 06:46:10.365602 1470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:10.366959 1470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:10.368160 1470 hybrid_clock.cc:648] HybridClock initialized: now 1786257970368116 us; error 35 us; skew 500 ppm
I20260809 06:46:10.368506 1470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:10 dist-test-slave-jpj1 krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257970, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:10.374289 1470 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:10.375757 1470 webserver.cc:460] Webserver started at http://127.27.209.254:36435/ using document root <none> and password file <none>
I20260809 06:46:10.376456 1470 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:10.376569 1470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:10.376869 1470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:10.379827 1470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "cd9e23c3340144629d936c86254f0c69"
format_stamp: "Formatted at 2026-08-09 06:46:10 on dist-test-slave-jpj1"
server_key: "d6ea4c887bb65158d928c64dbc749bcb"
server_key_iv: "124106821dff7cfd72e541aac026fd37"
server_key_version: "encryptionkey@0"
I20260809 06:46:10.380607 1470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "cd9e23c3340144629d936c86254f0c69"
format_stamp: "Formatted at 2026-08-09 06:46:10 on dist-test-slave-jpj1"
server_key: "d6ea4c887bb65158d928c64dbc749bcb"
server_key_iv: "124106821dff7cfd72e541aac026fd37"
server_key_version: "encryptionkey@0"
I20260809 06:46:10.385517 1470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:46:10.388777 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:10.390017 1470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:46:10.390169 1470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "cd9e23c3340144629d936c86254f0c69"
format_stamp: "Formatted at 2026-08-09 06:46:10 on dist-test-slave-jpj1"
server_key: "d6ea4c887bb65158d928c64dbc749bcb"
server_key_iv: "124106821dff7cfd72e541aac026fd37"
server_key_version: "encryptionkey@0"
I20260809 06:46:10.390363 1470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:10.412977 1470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:10.417029 1470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:10.417304 1470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:10.434324 1470 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:33313
I20260809 06:46:10.434356 1577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:33313 every 8 connection(s)
I20260809 06:46:10.435653 1470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:10.436218 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 1470
I20260809 06:46:10.436439 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:10.437173 28487 external_mini_cluster.cc:1428] Setting key fcc066a2519c7b72f302ec67965eb1e1
I20260809 06:46:10.439965 1578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:46:10 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:10.450294 1578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69: Bootstrap starting.
I20260809 06:46:10.453486 1578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:10.454449 1578 log.cc:824] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:10.457105 1578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69: No bootstrap required, opened a new log
I20260809 06:46:10.457749 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:10.439098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.27.209.254:33313)
0809 06:46:10.439895 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:10.439906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:10.439947 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:46:10.441593 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:10.441614 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:10.441977 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:10.442337 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:10.443803 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:10.445114 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:10.446320 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:10.446950 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:10.447134 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:10.451075 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:10.451083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:10.451097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:10.451128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:10.453974 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:10.454591 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:10.454597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:10.454603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:10.454765 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:10.455121 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:10.455125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:10.455127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:10.455695 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:10.455961 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:10.456379 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 06:46:10.456746 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":596,"thread_start_us":374,"threads_started":1}
I20260809 06:46:10.465214 1578 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9e23c3340144629d936c86254f0c69" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 33313 } }
I20260809 06:46:10.465559 1578 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:10.465654 1578 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd9e23c3340144629d936c86254f0c69, State: Initialized, Role: FOLLOWER
I20260809 06:46:10.466203 1578 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9e23c3340144629d936c86254f0c69" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 33313 } }
I20260809 06:46:10.466414 1578 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:10.466539 1578 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:10.466691 1578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:10.469079 1578 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9e23c3340144629d936c86254f0c69" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 33313 } }
I20260809 06:46:10.469554 1578 leader_election.cc:302] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd9e23c3340144629d936c86254f0c69; no voters:
I20260809 06:46:10.470093 1578 leader_election.cc:288] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:10.470208 1584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:10.470800 1584 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [term 1 LEADER]: Becoming Leader. State: Replica: cd9e23c3340144629d936c86254f0c69, State: Running, Role: LEADER
I20260809 06:46:10.471354 1584 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9e23c3340144629d936c86254f0c69" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 33313 } }
I20260809 06:46:10.472021 1578 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:10.473765 1586 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd9e23c3340144629d936c86254f0c69. Latest consensus state: current_term: 1 leader_uuid: "cd9e23c3340144629d936c86254f0c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9e23c3340144629d936c86254f0c69" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 33313 } } }
I20260809 06:46:10.474116 1586 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:10.474587 1585 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd9e23c3340144629d936c86254f0c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd9e23c3340144629d936c86254f0c69" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 33313 } } }
I20260809 06:46:10.474907 1585 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:10.475162 1592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:10.481231 1592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:10.489236 1592 catalog_manager.cc:1347] Generated new cluster ID: 927889c3c34d4c29a634b57ecb489910
I20260809 06:46:10.489352 1592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:10.499629 1592 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:10.500842 1592 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:10.506631 1592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd9e23c3340144629d936c86254f0c69: Generated new TSK 0
I20260809 06:46:10.507365 1592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:46:10.565310 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:33313
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:10.886404 1609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:10.886672 1609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:10.886811 1609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:10.893973 1609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:10.894081 1609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:10.894235 1609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:10.903975 1609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:10.905288 1609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:10.906595 1609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:10.919062 1619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:10.919241 1618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:10.919039 1621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:10.919842 1609 server_base.cc:1034] running on GCE node
I20260809 06:46:10.920409 1609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:10.921283 1609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:10.922462 1609 hybrid_clock.cc:648] HybridClock initialized: now 1786257970922411 us; error 46 us; skew 500 ppm
I20260809 06:46:10.922760 1609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:10 dist-test-slave-jpj1 krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257970, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:10.925828 1609 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:46:10.927141 1609 webserver.cc:460] Webserver started at http://127.27.209.193:33749/ using document root <none> and password file <none>
I20260809 06:46:10.927839 1609 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:10.927955 1609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:10.928251 1609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:10.931051 1609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "894bb2e1c69d4b6dad4fd03656f92edc"
format_stamp: "Formatted at 2026-08-09 06:46:10 on dist-test-slave-jpj1"
server_key: "80f7150d8dca5ab3daf3c41c0cdac141"
server_key_iv: "ea0c77ca555636aa5170c20fa65cd2f0"
server_key_version: "encryptionkey@0"
I20260809 06:46:10.931828 1609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "894bb2e1c69d4b6dad4fd03656f92edc"
format_stamp: "Formatted at 2026-08-09 06:46:10 on dist-test-slave-jpj1"
server_key: "80f7150d8dca5ab3daf3c41c0cdac141"
server_key_iv: "ea0c77ca555636aa5170c20fa65cd2f0"
server_key_version: "encryptionkey@0"
I20260809 06:46:10.936622 1609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:46:10.939906 1633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:10.941135 1609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:10.941277 1609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "894bb2e1c69d4b6dad4fd03656f92edc"
format_stamp: "Formatted at 2026-08-09 06:46:10 on dist-test-slave-jpj1"
server_key: "80f7150d8dca5ab3daf3c41c0cdac141"
server_key_iv: "ea0c77ca555636aa5170c20fa65cd2f0"
server_key_version: "encryptionkey@0"
I20260809 06:46:10.941452 1609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:10.970968 1609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:10.975184 1609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:10.975456 1609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:10.976425 1609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:10.977815 1609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:10.977896 1609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:10.978022 1609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:10.978093 1609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.008656 1609 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:33843
I20260809 06:46:11.008759 1801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:33843 every 8 connection(s)
I20260809 06:46:11.010010 1609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:11.014597 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 1609
I20260809 06:46:11.014833 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:11.015447 28487 external_mini_cluster.cc:1428] Setting key aadd3f27a7e07099f0d9ee3626f0eb6b
Aug 09 06:46:11 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257970, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:11.030664 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:11.012609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:56975 (local address 127.27.209.254:33313)
0809 06:46:11.013006 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:11.013010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:11.013994 (+ 984us) server_negotiation.cc:407] Connection header received
0809 06:46:11.015947 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:11.015953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:11.016046 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:11.016159 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:11.018105 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.019092 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.020680 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.021221 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.021350 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:11.024818 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:11.024824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:11.024828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:11.024867 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:11.027366 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:11.028277 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:11.028281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:11.028283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:11.028412 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:11.028872 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:11.028875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:11.028877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:11.029149 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:11.029271 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:11.030194 (+ 923us) server_negotiation.cc:299] Negotiation successful
0809 06:46:11.030385 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":193,"threads_started":1}
I20260809 06:46:11.032052 1806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:11.013171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:33313 (local address 127.27.209.193:56975)
0809 06:46:11.013840 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:11.013877 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:11.015658 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:11.016439 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:11.016448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:11.017190 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:11.017909 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:11.017928 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.019235 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.019238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:11.020533 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:11.020546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.021413 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.021417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:11.021622 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:11.022261 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:11.022283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:11.024587 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:11.027512 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:11.027521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:11.027567 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:11.028164 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:11.028512 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:11.028517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:11.028520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:11.028767 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:11.029278 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:11.029287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:11.029780 (+ 493us) client_negotiation.cc:769] Sending connection context
0809 06:46:11.030079 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 06:46:11.030612 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":261,"threads_started":1}
I20260809 06:46:11.033826 1802 heartbeater.cc:347] Connected to a master server at 127.27.209.254:33313
I20260809 06:46:11.034137 1802 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:11.034927 1802 heartbeater.cc:511] Master 127.27.209.254:33313 requested a full tablet report, sending...
I20260809 06:46:11.037434 1522 ts_manager.cc:194] Registered new tserver with Master: 894bb2e1c69d4b6dad4fd03656f92edc (127.27.209.193:33843)
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
I20260809 06:46:11.039134 1522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:56975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:46:11.066872 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:33313
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:11.391808 1809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:11.392108 1809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:11.392261 1809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:11.399621 1809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:11.399742 1809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:11.399891 1809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:11.411229 1809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:11.412537 1809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:11.413856 1809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:11.426368 1815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:11.426332 1821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:11.426402 1816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:11.427259 1809 server_base.cc:1034] running on GCE node
I20260809 06:46:11.427977 1809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:11.428851 1809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:11.430040 1809 hybrid_clock.cc:648] HybridClock initialized: now 1786257971429988 us; error 42 us; skew 500 ppm
I20260809 06:46:11.430330 1809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:11 dist-test-slave-jpj1 krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257971, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:11.433338 1809 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:46:11.434913 1809 webserver.cc:460] Webserver started at http://127.27.209.194:42769/ using document root <none> and password file <none>
I20260809 06:46:11.435599 1809 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:11.435715 1809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:11.436010 1809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:11.438716 1809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "70be8614c59a4fd2baf7cefbb1b302ae"
format_stamp: "Formatted at 2026-08-09 06:46:11 on dist-test-slave-jpj1"
server_key: "2ddb98639ff3f600b87c034ca7a567a5"
server_key_iv: "a57265123cae83e0059f7ffda4d22405"
server_key_version: "encryptionkey@0"
I20260809 06:46:11.439459 1809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "70be8614c59a4fd2baf7cefbb1b302ae"
format_stamp: "Formatted at 2026-08-09 06:46:11 on dist-test-slave-jpj1"
server_key: "2ddb98639ff3f600b87c034ca7a567a5"
server_key_iv: "a57265123cae83e0059f7ffda4d22405"
server_key_version: "encryptionkey@0"
I20260809 06:46:11.444393 1809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:46:11.447898 1830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.449110 1809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:11.449258 1809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "70be8614c59a4fd2baf7cefbb1b302ae"
format_stamp: "Formatted at 2026-08-09 06:46:11 on dist-test-slave-jpj1"
server_key: "2ddb98639ff3f600b87c034ca7a567a5"
server_key_iv: "a57265123cae83e0059f7ffda4d22405"
server_key_version: "encryptionkey@0"
I20260809 06:46:11.449430 1809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:11.483278 1809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:11.487150 1809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:11.487411 1809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:11.488374 1809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:11.489677 1809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:11.489754 1809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.489889 1809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:11.489945 1809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.519804 1809 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:44121
I20260809 06:46:11.519845 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:44121 every 8 connection(s)
I20260809 06:46:11.521164 1809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:11.526409 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 1809
I20260809 06:46:11.526580 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:11.526997 28487 external_mini_cluster.cc:1428] Setting key 07f1b249b5d9dc2a925629668d8f4d8f
Aug 09 06:46:11 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257971, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:11.541203 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:11.523586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:39075 (local address 127.27.209.254:33313)
0809 06:46:11.523761 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:11.523765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:11.524875 (+ 1110us) server_negotiation.cc:407] Connection header received
0809 06:46:11.526690 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:11.526696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:11.526782 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:11.526869 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:11.528630 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.529624 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.531263 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.531856 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.531976 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:11.535507 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:11.535513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:11.535516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:11.535569 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:11.538224 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:11.538935 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:11.538940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:11.538943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:11.539092 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:11.539492 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:11.539497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:11.539500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:11.539867 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:11.540008 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:11.540640 (+ 632us) server_negotiation.cc:299] Negotiation successful
0809 06:46:11.540879 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:46:11.542083 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:11.524058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:33313 (local address 127.27.209.194:39075)
0809 06:46:11.524709 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:11.524754 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:11.526405 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:11.527126 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:11.527137 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:11.527751 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:11.528463 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:11.528489 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.529805 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.529809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:11.531111 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:11.531124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:11.532067 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:11.532072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:11.532268 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:11.532905 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:11.532932 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:11.535299 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:11.538374 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:11.538381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:11.538393 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:11.538817 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:11.539185 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:11.539188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:11.539190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:11.539383 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:11.539977 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:11.539983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:11.540304 (+ 321us) client_negotiation.cc:769] Sending connection context
0809 06:46:11.540538 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 06:46:11.540906 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
I20260809 06:46:11.543475 1992 heartbeater.cc:347] Connected to a master server at 127.27.209.254:33313
I20260809 06:46:11.543794 1992 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:11.544550 1992 heartbeater.cc:511] Master 127.27.209.254:33313 requested a full tablet report, sending...
I20260809 06:46:11.546087 1522 ts_manager.cc:194] Registered new tserver with Master: 70be8614c59a4fd2baf7cefbb1b302ae (127.27.209.194:44121)
I20260809 06:46:11.546856 1522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:39075
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:46:11.574080 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:33313
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:11.886950 1999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:11.887238 1999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:11.887384 1999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:11.894575 1999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:11.894703 1999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:11.894886 1999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:11.904991 1999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:11.906311 1999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:11.907713 1999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:11.920275 2005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:11.920320 2006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:11.920477 2008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:11.921064 1999 server_base.cc:1034] running on GCE node
I20260809 06:46:11.921679 1999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:11.922574 1999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:11.923766 1999 hybrid_clock.cc:648] HybridClock initialized: now 1786257971923725 us; error 36 us; skew 500 ppm
I20260809 06:46:11.924063 1999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:11 dist-test-slave-jpj1 krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257971, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:11.927027 1999 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:46:11.928462 1999 webserver.cc:460] Webserver started at http://127.27.209.195:36647/ using document root <none> and password file <none>
I20260809 06:46:11.929131 1999 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:11.929246 1999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:11.929549 1999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:11.932415 1999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "e75390248e184a318024cf0559c2b41e"
format_stamp: "Formatted at 2026-08-09 06:46:11 on dist-test-slave-jpj1"
server_key: "22e6dc0677602e43fa36224ab15326bf"
server_key_iv: "71a591e3a9ec84b267e5286e91555f89"
server_key_version: "encryptionkey@0"
I20260809 06:46:11.933157 1999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "e75390248e184a318024cf0559c2b41e"
format_stamp: "Formatted at 2026-08-09 06:46:11 on dist-test-slave-jpj1"
server_key: "22e6dc0677602e43fa36224ab15326bf"
server_key_iv: "71a591e3a9ec84b267e5286e91555f89"
server_key_version: "encryptionkey@0"
I20260809 06:46:11.938146 1999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 06:46:11.941408 2017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.942670 1999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:46:11.942818 1999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "e75390248e184a318024cf0559c2b41e"
format_stamp: "Formatted at 2026-08-09 06:46:11 on dist-test-slave-jpj1"
server_key: "22e6dc0677602e43fa36224ab15326bf"
server_key_iv: "71a591e3a9ec84b267e5286e91555f89"
server_key_version: "encryptionkey@0"
I20260809 06:46:11.942996 1999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:11.961866 1999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:11.965845 1999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:11.966097 1999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:11.966976 1999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:11.968360 1999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:11.968441 1999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.968562 1999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:11.968616 1999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:11.998978 1999 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:40933
I20260809 06:46:11.999019 2178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:40933 every 8 connection(s)
I20260809 06:46:12.000240 1999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:12.001178 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 1999
I20260809 06:46:12.001374 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:12.001909 28487 external_mini_cluster.cc:1428] Setting key 08ccf62c5d4a0469d01c08609b790c95
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257971, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:12.021020 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.003117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:50005 (local address 127.27.209.254:33313)
0809 06:46:12.003322 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:12.003328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:12.004546 (+ 1218us) server_negotiation.cc:407] Connection header received
0809 06:46:12.006488 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:12.006494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:12.006607 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:12.006693 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:12.008521 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.009256 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.011454 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.011908 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.012023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.015292 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:12.015299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:12.015302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:12.015334 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:12.018148 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.018823 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.018829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.018833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.018952 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.019343 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.019348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.019352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.019679 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:12.019818 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:12.020457 (+ 639us) server_negotiation.cc:299] Negotiation successful
0809 06:46:12.020679 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:46:12.021871 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.003653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:33313 (local address 127.27.209.195:50005)
0809 06:46:12.004404 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:12.004444 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:12.006232 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:12.006949 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:12.006958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:12.007571 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:12.008353 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:12.008381 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.009411 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.009417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:12.011315 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:12.011330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.012132 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.012139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:12.012367 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.013055 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:12.013081 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:12.015109 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:12.018273 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:12.018283 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:12.018306 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:12.018705 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:12.019055 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:12.019060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:12.019063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:12.019237 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:12.019795 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:12.019803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:12.020118 (+ 315us) client_negotiation.cc:769] Sending connection context
0809 06:46:12.020364 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 06:46:12.020850 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":282,"threads_started":1}
I20260809 06:46:12.023126 2179 heartbeater.cc:347] Connected to a master server at 127.27.209.254:33313
I20260809 06:46:12.023420 2179 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:12.024133 2179 heartbeater.cc:511] Master 127.27.209.254:33313 requested a full tablet report, sending...
I20260809 06:46:12.025503 1522 ts_manager.cc:194] Registered new tserver with Master: e75390248e184a318024cf0559c2b41e (127.27.209.195:40933)
I20260809 06:46:12.026232 1522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:50005
I20260809 06:46:12.029381 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:12.042105 1802 heartbeater.cc:503] Master 127.27.209.254:33313 was elected leader, sending a full tablet report...
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[1453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:12.081444 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.066269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.27.209.193:33843)
0809 06:46:12.066901 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:12.066909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:12.066946 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:46:12.067010 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:12.067017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:12.067229 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:12.067376 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:12.069005 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.070045 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.071231 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.071871 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.071968 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.075968 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:12.075981 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:12.075998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:12.076039 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:12.078834 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.079493 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.079501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.079503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.079712 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.080048 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.080051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.080053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.080396 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:12.080638 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:12.080878 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 06:46:12.081080 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":322,"threads_started":1}
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:12.098503 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.085570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.27.209.254:33313)
0809 06:46:12.085808 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:12.085814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:12.085894 (+ 80us) server_negotiation.cc:407] Connection header received
0809 06:46:12.086057 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:12.086061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:12.086124 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:12.086209 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:12.087379 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.088148 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.089362 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.089802 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.089913 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.093483 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:12.093488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:12.093492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:12.093523 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:12.096066 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.096646 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.096657 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.096659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.096806 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.097154 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.097159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.097161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.097480 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:12.097709 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:12.097989 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 06:46:12.098213 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260809 06:46:12.099102 1522 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37024
I20260809 06:46:12.100162 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1609
I20260809 06:46:12.112339 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.101054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.27.209.193:33843)
0809 06:46:12.101262 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:12.101268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:12.101404 (+ 136us) server_negotiation.cc:407] Connection header received
0809 06:46:12.101558 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:12.101563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:12.101663 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:12.101747 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:12.102972 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.103709 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.104891 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.105341 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.105450 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.107571 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:12.107579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:12.107584 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:12.107624 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:12.109977 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.110559 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.110565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.110568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.110694 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.111035 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.111040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.111044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.111284 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:12.111503 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:12.111823 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 06:46:12.112032 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:46:12.112980 1793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:12.196486 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 1609
I20260809 06:46:12.216497 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1809
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:46:12.232028 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.217487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.27.209.194:44121)
0809 06:46:12.218036 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:12.218043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:12.218070 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:46:12.218137 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:12.218144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:12.218353 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:12.218489 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:12.219649 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.220676 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.221866 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.222416 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.222514 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.226570 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:12.226580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:12.226595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:12.226630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:12.229467 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.230104 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.230112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.230114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.230247 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.230548 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.230551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.230553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.230889 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:12.231122 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:12.231383 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 06:46:12.231621 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":289,"threads_started":1}
I20260809 06:46:12.233278 1983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:12.314242 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 1809
I20260809 06:46:12.337832 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1999
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[1453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:46:12.353811 2197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:12.338776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.27.209.195:40933)
0809 06:46:12.339391 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:12.339398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:12.339430 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:46:12.339492 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:12.339499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:12.339781 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:12.339907 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:12.341405 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.342485 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.343966 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:12.344529 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:12.344627 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:12.348736 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:12.348748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:12.348763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:12.348796 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:12.351334 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.352033 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.352041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.352043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.352184 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:12.352479 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:12.352481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:12.352483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:12.352806 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:12.353042 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:12.353259 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 06:46:12.353459 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":273,"threads_started":1}
I20260809 06:46:12.355062 2173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:12.436632 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 1999
I20260809 06:46:12.461154 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 1470
I20260809 06:46:12.462414 1570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:12.564376 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 1470
2026-08-09T06:46:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3942 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[2207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[2207](info): set up 2 sockets
Aug 09 06:46:12 dist-test-slave-jpj1 krb5kdc[2207](info): commencing operation
krb5kdc: starting...
W20260809 06:46:13.753796 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:13 dist-test-slave-jpj1 krb5kdc[2207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:46:13.955655 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40937
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:40937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:14.282227 2227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:14.282536 2227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:14.282676 2227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:14.289901 2227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:14.290021 2227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:14.290102 2227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:14.290179 2227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:14.290246 2227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:14.300006 2227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:40937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:14.301342 2227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:14.302652 2227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:14.313946 2237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:14.314016 2236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:14.313987 2239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:14.314771 2227 server_base.cc:1034] running on GCE node
I20260809 06:46:14.315604 2227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:14.317009 2227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:14.318224 2227 hybrid_clock.cc:648] HybridClock initialized: now 1786257974318171 us; error 44 us; skew 500 ppm
I20260809 06:46:14.318559 2227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:14 dist-test-slave-jpj1 krb5kdc[2207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257974, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:14.321684 2227 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:14.322993 2227 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:14.323114 2227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:14.323446 2227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:14.326552 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "07ed8785d45645e79c37500671a9cfd9"
format_stamp: "Formatted at 2026-08-09 06:46:14 on dist-test-slave-jpj1"
server_key: "27ecee93b8d334a994a751ed6edc88b7"
server_key_iv: "fb5849bb2374844019703a7d5b7006f9"
server_key_version: "encryptionkey@0"
I20260809 06:46:14.327382 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "07ed8785d45645e79c37500671a9cfd9"
format_stamp: "Formatted at 2026-08-09 06:46:14 on dist-test-slave-jpj1"
server_key: "27ecee93b8d334a994a751ed6edc88b7"
server_key_iv: "fb5849bb2374844019703a7d5b7006f9"
server_key_version: "encryptionkey@0"
I20260809 06:46:14.332710 2227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:46:14.336193 2246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:14.337461 2227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 06:46:14.337630 2227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "07ed8785d45645e79c37500671a9cfd9"
format_stamp: "Formatted at 2026-08-09 06:46:14 on dist-test-slave-jpj1"
server_key: "27ecee93b8d334a994a751ed6edc88b7"
server_key_iv: "fb5849bb2374844019703a7d5b7006f9"
server_key_version: "encryptionkey@0"
I20260809 06:46:14.337844 2227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:14.351318 2227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:14.355273 2227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:14.355564 2227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:14.372844 2227 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:40937
I20260809 06:46:14.372865 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:40937 every 8 connection(s)
I20260809 06:46:14.373720 2227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:14.376431 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 2227
I20260809 06:46:14.376672 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:14.377470 28487 external_mini_cluster.cc:1428] Setting key 0dc6c4b992f91e83be8d7bc744f6a29d
I20260809 06:46:14.378026 2322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:14.392365 2322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9: Bootstrap starting.
Aug 09 06:46:14 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:14.395864 2322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:14.396947 2322 log.cc:824] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:14.400270 2322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9: No bootstrap required, opened a new log
I20260809 06:46:14.401618 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:14.379605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.27.209.254:40937)
0809 06:46:14.380384 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:14.380395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:14.380436 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:46:14.382683 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:14.382706 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:14.383084 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:14.383407 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:14.385286 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:14.386564 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:14.388368 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:14.389097 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:14.389243 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:14.393892 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:14.393900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:14.393913 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:14.393949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:14.397004 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:14.397812 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:14.397819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:14.397826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:14.398000 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:14.398414 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:14.398419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:14.398422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:14.399145 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:14.399449 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:14.399832 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 06:46:14.400276 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260809 06:46:14.407991 2322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ed8785d45645e79c37500671a9cfd9" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40937 } }
I20260809 06:46:14.408324 2322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:14.408397 2322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ed8785d45645e79c37500671a9cfd9, State: Initialized, Role: FOLLOWER
I20260809 06:46:14.408927 2322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ed8785d45645e79c37500671a9cfd9" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40937 } }
I20260809 06:46:14.409138 2322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:14.409240 2322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:14.409360 2322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:14.411461 2322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ed8785d45645e79c37500671a9cfd9" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40937 } }
I20260809 06:46:14.411984 2322 leader_election.cc:302] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07ed8785d45645e79c37500671a9cfd9; no voters:
I20260809 06:46:14.412492 2322 leader_election.cc:288] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:14.412616 2330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:14.413671 2330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [term 1 LEADER]: Becoming Leader. State: Replica: 07ed8785d45645e79c37500671a9cfd9, State: Running, Role: LEADER
I20260809 06:46:14.414240 2330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ed8785d45645e79c37500671a9cfd9" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40937 } }
I20260809 06:46:14.414496 2322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:14.417080 2332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07ed8785d45645e79c37500671a9cfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ed8785d45645e79c37500671a9cfd9" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40937 } } }
I20260809 06:46:14.417053 2335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07ed8785d45645e79c37500671a9cfd9. Latest consensus state: current_term: 1 leader_uuid: "07ed8785d45645e79c37500671a9cfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07ed8785d45645e79c37500671a9cfd9" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40937 } } }
I20260809 06:46:14.417449 2332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:14.417454 2335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:14.419014 2340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:14.424593 2340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:14.433703 2340 catalog_manager.cc:1347] Generated new cluster ID: b7404364ffe6499eb19c89c712099cd4
I20260809 06:46:14.433810 2340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:14.454564 2340 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:14.455883 2340 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:14.466387 2340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9: Generated new TSK 0
I20260809 06:46:14.467180 2340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:46:14.526261 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40937
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:14.860482 2366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:14.860773 2366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:14.860909 2366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:14.867962 2366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:14.868073 2366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:14.868225 2366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:14.877493 2366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:14.878720 2366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:14.880038 2366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:14.892596 2373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:14.892545 2376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:14.892515 2374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:14.893319 2366 server_base.cc:1034] running on GCE node
I20260809 06:46:14.893901 2366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:14.894757 2366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:14.895960 2366 hybrid_clock.cc:648] HybridClock initialized: now 1786257974895902 us; error 51 us; skew 500 ppm
I20260809 06:46:14.896255 2366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:14 dist-test-slave-jpj1 krb5kdc[2207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257974, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:14.899227 2366 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:46:14.900302 2366 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:14.900406 2366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:14.900699 2366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:14.903717 2366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "370edeec11264dceb200fcba51f6a78b"
format_stamp: "Formatted at 2026-08-09 06:46:14 on dist-test-slave-jpj1"
server_key: "888962ca7ba53aa5140251586f438e81"
server_key_iv: "a56e25b0df93d7cbc6940303c579e5e4"
server_key_version: "encryptionkey@0"
I20260809 06:46:14.904515 2366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "370edeec11264dceb200fcba51f6a78b"
format_stamp: "Formatted at 2026-08-09 06:46:14 on dist-test-slave-jpj1"
server_key: "888962ca7ba53aa5140251586f438e81"
server_key_iv: "a56e25b0df93d7cbc6940303c579e5e4"
server_key_version: "encryptionkey@0"
I20260809 06:46:14.909462 2366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:46:14.912897 2384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:14.914132 2366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:46:14.914291 2366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "370edeec11264dceb200fcba51f6a78b"
format_stamp: "Formatted at 2026-08-09 06:46:14 on dist-test-slave-jpj1"
server_key: "888962ca7ba53aa5140251586f438e81"
server_key_iv: "a56e25b0df93d7cbc6940303c579e5e4"
server_key_version: "encryptionkey@0"
I20260809 06:46:14.914541 2366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:14.931118 2366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:14.935209 2366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:14.935459 2366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:14.936430 2366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:14.937711 2366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:14.937821 2366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:14.937927 2366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:14.938002 2366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:14.969063 2366 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:35973
I20260809 06:46:14.969118 2549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:35973 every 8 connection(s)
I20260809 06:46:14.969913 2366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:14.979555 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 2366
I20260809 06:46:14.979770 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:14.980580 28487 external_mini_cluster.cc:1428] Setting key a2a348e0518f108f3e287b724569a4ab
Aug 09 06:46:14 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257974, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:14.989219 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:14.972263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:58239 (local address 127.27.209.254:40937)
0809 06:46:14.972657 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:14.972661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:14.973677 (+ 1016us) server_negotiation.cc:407] Connection header received
0809 06:46:14.975418 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:14.975423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:14.975489 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:14.975636 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:14.977013 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:14.977739 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:14.979188 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:14.979661 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:14.979779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:14.983746 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:14.983750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:14.983753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:14.983784 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:14.986214 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:14.986937 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:14.986942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:14.986945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:14.987067 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:14.987474 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:14.987479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:14.987482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:14.987800 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:14.987934 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:14.988636 (+ 702us) server_negotiation.cc:299] Negotiation successful
0809 06:46:14.988869 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":198,"threads_started":1}
I20260809 06:46:14.990119 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:14.972845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40937 (local address 127.27.209.193:58239)
0809 06:46:14.973518 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:14.973556 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:14.975166 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:14.975815 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:14.975823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:14.976311 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:14.976854 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:14.976870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:14.977856 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:14.977859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:14.979050 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:14.979062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:14.979910 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:14.979920 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:14.980192 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:14.980925 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:14.980952 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:14.983485 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:14.986372 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:14.986381 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:14.986398 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:14.986817 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:14.987173 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:14.987178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:14.987182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:14.987366 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:14.987926 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:14.987934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:14.988258 (+ 324us) client_negotiation.cc:769] Sending connection context
0809 06:46:14.988501 (+ 243us) client_negotiation.cc:240] Negotiation successful
0809 06:46:14.989018 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":267,"threads_started":1}
I20260809 06:46:14.991495 2550 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40937
I20260809 06:46:14.991822 2550 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:14.992456 2550 heartbeater.cc:511] Master 127.27.209.254:40937 requested a full tablet report, sending...
I20260809 06:46:14.994462 2273 ts_manager.cc:194] Registered new tserver with Master: 370edeec11264dceb200fcba51f6a78b (127.27.209.193:35973)
I20260809 06:46:14.995924 2273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:58239
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:46:15.027046 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40937
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:15.351302 2560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:15.351606 2560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:15.351742 2560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:15.358920 2560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:15.359027 2560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:15.359176 2560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:15.369019 2560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:15.370303 2560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:15.371593 2560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:15.385380 2565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:15.385422 2566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:15.385377 2569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:15.387096 2560 server_base.cc:1034] running on GCE node
I20260809 06:46:15.387787 2560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:15.388823 2560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:15.390053 2560 hybrid_clock.cc:648] HybridClock initialized: now 1786257975389982 us; error 59 us; skew 500 ppm
I20260809 06:46:15.390415 2560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:15 dist-test-slave-jpj1 krb5kdc[2207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257975, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:15.396229 2560 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:46:15.397382 2560 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:15.397478 2560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:15.397794 2560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:15.400817 2560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7"
format_stamp: "Formatted at 2026-08-09 06:46:15 on dist-test-slave-jpj1"
server_key: "c4024b2b013f3c3288064884640c43b7"
server_key_iv: "0ccbd90ba80fc36aa4070f5d34a9e791"
server_key_version: "encryptionkey@0"
I20260809 06:46:15.401641 2560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7"
format_stamp: "Formatted at 2026-08-09 06:46:15 on dist-test-slave-jpj1"
server_key: "c4024b2b013f3c3288064884640c43b7"
server_key_iv: "0ccbd90ba80fc36aa4070f5d34a9e791"
server_key_version: "encryptionkey@0"
I20260809 06:46:15.407220 2560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:46:15.411151 2576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:15.412504 2560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:15.412652 2560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7"
format_stamp: "Formatted at 2026-08-09 06:46:15 on dist-test-slave-jpj1"
server_key: "c4024b2b013f3c3288064884640c43b7"
server_key_iv: "0ccbd90ba80fc36aa4070f5d34a9e791"
server_key_version: "encryptionkey@0"
I20260809 06:46:15.412849 2560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:15.449604 2560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:15.453531 2560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:15.453778 2560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:15.454782 2560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:15.456128 2560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:15.456223 2560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:15.456331 2560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:15.456408 2560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:15.486382 2560 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:35673
I20260809 06:46:15.486452 2726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:35673 every 8 connection(s)
I20260809 06:46:15.487193 2560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:15.487784 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 2560
I20260809 06:46:15.487980 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:15.488533 28487 external_mini_cluster.cc:1428] Setting key ee2861012b151618a22c62ae4e26699d
Aug 09 06:46:15 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257975, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:15.508260 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:15.489847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:39139 (local address 127.27.209.254:40937)
0809 06:46:15.490318 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:15.490324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:15.491487 (+ 1163us) server_negotiation.cc:407] Connection header received
0809 06:46:15.494018 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:15.494022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:15.494084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:15.494176 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:15.496096 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.496844 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.498902 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.499315 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.499422 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:15.502871 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:15.502876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:15.502879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:15.502912 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:15.505343 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:15.506030 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:15.506033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:15.506036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:15.506157 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:15.506560 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:15.506563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:15.506565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:15.506824 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:15.506954 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:15.507665 (+ 711us) server_negotiation.cc:299] Negotiation successful
0809 06:46:15.507881 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260809 06:46:15.509217 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:15.490465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40937 (local address 127.27.209.194:39139)
0809 06:46:15.491296 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:15.491357 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:15.493721 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:15.494437 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:15.494447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:15.495141 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:15.495931 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:15.495956 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.497003 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.497008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:15.498763 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:15.498778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.499558 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.499566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:15.499778 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:15.500386 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:15.500408 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:15.502649 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:15.505481 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:15.505490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:15.505506 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:15.505911 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:15.506267 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:15.506272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:15.506276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:15.506449 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:15.506948 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:15.506955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:15.507277 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 06:46:15.507515 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 06:46:15.508023 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
I20260809 06:46:15.510586 2727 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40937
I20260809 06:46:15.510896 2727 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:15.511514 2727 heartbeater.cc:511] Master 127.27.209.254:40937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
I20260809 06:46:15.513022 2273 ts_manager.cc:194] Registered new tserver with Master: 3b0b3dbc984a4b9a917d23a1ed4550b7 (127.27.209.194:35673)
I20260809 06:46:15.513859 2273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:39139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:46:15.536423 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40937
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:15.858619 2734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:15.858906 2734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:15.859031 2734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:15.866286 2734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:15.866398 2734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:15.866547 2734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:15.876420 2734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:15.877730 2734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:15.879081 2734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:15.891490 2740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:15.891520 2741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:15.891564 2743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:15.892246 2734 server_base.cc:1034] running on GCE node
I20260809 06:46:15.892848 2734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:15.893734 2734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:15.894936 2734 hybrid_clock.cc:648] HybridClock initialized: now 1786257975894888 us; error 42 us; skew 500 ppm
I20260809 06:46:15.895246 2734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:15 dist-test-slave-jpj1 krb5kdc[2207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257975, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:15.898201 2734 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:46:15.899374 2734 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:15.899477 2734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:15.899825 2734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:15.902710 2734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "3843f75ce85d493797c776fc3d654dc6"
format_stamp: "Formatted at 2026-08-09 06:46:15 on dist-test-slave-jpj1"
server_key: "b3e78cc9cc164d429d4833975ba061aa"
server_key_iv: "15cb8a83c04d3a8667dd430aedded44c"
server_key_version: "encryptionkey@0"
I20260809 06:46:15.903499 2734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "3843f75ce85d493797c776fc3d654dc6"
format_stamp: "Formatted at 2026-08-09 06:46:15 on dist-test-slave-jpj1"
server_key: "b3e78cc9cc164d429d4833975ba061aa"
server_key_iv: "15cb8a83c04d3a8667dd430aedded44c"
server_key_version: "encryptionkey@0"
I20260809 06:46:15.908581 2734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:46:15.911888 2750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:15.913244 2734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:15.913399 2734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "3843f75ce85d493797c776fc3d654dc6"
format_stamp: "Formatted at 2026-08-09 06:46:15 on dist-test-slave-jpj1"
server_key: "b3e78cc9cc164d429d4833975ba061aa"
server_key_iv: "15cb8a83c04d3a8667dd430aedded44c"
server_key_version: "encryptionkey@0"
I20260809 06:46:15.913599 2734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:15.947279 2734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:15.951357 2734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:15.951634 2734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:15.952608 2734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:15.953960 2734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:15.954049 2734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:15.954164 2734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:15.954244 2734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:15.985421 2734 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:44873
I20260809 06:46:15.985479 2918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:44873 every 8 connection(s)
I20260809 06:46:15.986244 2734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:15.986652 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 2734
I20260809 06:46:15.986853 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:15.987383 28487 external_mini_cluster.cc:1428] Setting key 99cda6e3e63c6768b76219bd718a4b80
Aug 09 06:46:15 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257975, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:15.999326 2550 heartbeater.cc:503] Master 127.27.209.254:40937 was elected leader, sending a full tablet report...
I20260809 06:46:16.005522 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:15.988809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:51887 (local address 127.27.209.254:40937)
0809 06:46:15.989011 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:15.989017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:15.990493 (+ 1476us) server_negotiation.cc:407] Connection header received
0809 06:46:15.992417 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:15.992421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:15.992487 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:15.992572 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:15.993990 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.994714 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.996246 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.996650 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.996770 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:15.999915 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:15.999922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:15.999926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:15.999963 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:16.002910 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.003605 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.003608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.003610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.003734 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.004066 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.004068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.004070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.004348 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:16.004480 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.005119 (+ 639us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.005295 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:46:16.006497 2921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:15.989432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40937 (local address 127.27.209.195:51887)
0809 06:46:15.990313 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:15.990375 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:15.992154 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:15.992754 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:15.992762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:15.993277 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:15.993823 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:15.993841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.994829 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.994832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:15.996112 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:15.996125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:15.996818 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:15.996822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:15.997002 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:15.997574 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:15.997594 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:15.999722 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:16.003045 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:16.003052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:16.003064 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:16.003474 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:16.003814 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:16.003818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:16.003820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:16.003987 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:16.004497 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:16.004503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:16.004804 (+ 301us) client_negotiation.cc:769] Sending connection context
0809 06:46:16.005038 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 06:46:16.005394 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260809 06:46:16.007840 2919 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40937
I20260809 06:46:16.008111 2919 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:16.008725 2919 heartbeater.cc:511] Master 127.27.209.254:40937 requested a full tablet report, sending...
I20260809 06:46:16.010080 2273 ts_manager.cc:194] Registered new tserver with Master: 3843f75ce85d493797c776fc3d654dc6 (127.27.209.195:44873)
I20260809 06:46:16.010860 2273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:51887
I20260809 06:46:16.015054 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:16.032191 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.019849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.27.209.254:40937)
0809 06:46:16.020011 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.020016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.020509 (+ 493us) server_negotiation.cc:407] Connection header received
0809 06:46:16.020672 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.020677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.020747 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.020835 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:16.022278 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.023006 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.024294 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.024737 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.024852 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.027067 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:16.027072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:16.027075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:16.027107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:16.029619 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.030240 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.030245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.030249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.030385 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.030740 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.030745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.030748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.031067 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:16.031333 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.031657 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.031865 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:46:16.036087 2273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:16.042941 2273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:16.076206 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.055194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.27.209.193:35973)
0809 06:46:16.066924 (+ 11730us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.066933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.068056 (+ 1123us) server_negotiation.cc:407] Connection header received
0809 06:46:16.068180 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.068189 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.068444 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.068632 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:16.069372 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.070768 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.073593 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.075002 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.075198 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.075382 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.075674 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.075849 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11536,"thread_start_us":11374,"threads_started":1}
I20260809 06:46:16.076188 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.054895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:35973 (local address 127.0.0.1:57478)
0809 06:46:16.067323 (+ 12428us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:16.067406 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:16.067657 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:16.068663 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:16.068671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:16.068708 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:16.069122 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.069137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.070941 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.070946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.073444 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.073460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.075180 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.075186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.075389 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.075442 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 06:46:16.075615 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 06:46:16.075810 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12178,"thread_start_us":12055,"threads_started":1}
I20260809 06:46:16.076264 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.055644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44873 (local address 127.0.0.1:60032)
0809 06:46:16.067756 (+ 12112us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:16.067772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:16.067866 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:16.069068 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:16.069078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:16.069109 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:16.069955 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.069968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.071904 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.071908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.073380 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.073396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.075509 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.075513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.075700 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.075730 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:46:16.075813 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 06:46:16.075927 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11993,"thread_start_us":11829,"threads_started":1}
I20260809 06:46:16.077226 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.056047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.27.209.195:44873)
0809 06:46:16.068446 (+ 12399us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.068454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.068486 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:46:16.068581 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.068589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.068835 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.069284 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:16.070184 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.071782 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.073537 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.075002 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.075204 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.075382 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.076748 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.076911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12211,"thread_start_us":12046,"threads_started":1}
I20260809 06:46:16.077713 2944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.069025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:35673 (local address 127.0.0.1:49298)
0809 06:46:16.070476 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:16.070493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:16.070598 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:16.071379 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:16.071383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:16.071404 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:16.071878 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.071892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.073694 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.073699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.074932 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.074944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.076992 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.076998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.077153 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.077183 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:46:16.077283 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 06:46:16.077404 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":1204,"threads_started":1}
I20260809 06:46:16.078037 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.068523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.27.209.194:35673)
0809 06:46:16.069648 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.069656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.070586 (+ 930us) server_negotiation.cc:407] Connection header received
0809 06:46:16.070726 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.070733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.070953 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.071128 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:16.072093 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.073577 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.075079 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.076806 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.076982 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.077145 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.077562 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.077741 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":804,"threads_started":1}
I20260809 06:46:16.080854 2824 tablet_service.cc:1467] Processing CreateTablet for tablet e7b016ada4284411b6ec8c791f9b81d2 (DEFAULT_TABLE table=test-table [id=68324dcc60f8492baf539215b580db4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:16.080766 2458 tablet_service.cc:1467] Processing CreateTablet for tablet e7b016ada4284411b6ec8c791f9b81d2 (DEFAULT_TABLE table=test-table [id=68324dcc60f8492baf539215b580db4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:16.082154 2824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b016ada4284411b6ec8c791f9b81d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:16.082237 2458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b016ada4284411b6ec8c791f9b81d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:16.081795 2639 tablet_service.cc:1467] Processing CreateTablet for tablet e7b016ada4284411b6ec8c791f9b81d2 (DEFAULT_TABLE table=test-table [id=68324dcc60f8492baf539215b580db4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:16.083042 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b016ada4284411b6ec8c791f9b81d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:16.095607 2945 tablet_bootstrap.cc:492] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Bootstrap starting.
I20260809 06:46:16.096881 2948 tablet_bootstrap.cc:492] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Bootstrap starting.
I20260809 06:46:16.098851 2950 tablet_bootstrap.cc:492] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Bootstrap starting.
I20260809 06:46:16.099589 2945 tablet_bootstrap.cc:654] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:16.100345 2948 tablet_bootstrap.cc:654] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:16.100914 2945 log.cc:824] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:16.101573 2948 log.cc:824] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:16.101933 2950 tablet_bootstrap.cc:654] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:16.102963 2950 log.cc:824] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:16.104192 2945 tablet_bootstrap.cc:492] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: No bootstrap required, opened a new log
I20260809 06:46:16.104471 2945 ts_tablet_manager.cc:1397] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260809 06:46:16.104535 2948 tablet_bootstrap.cc:492] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: No bootstrap required, opened a new log
I20260809 06:46:16.104797 2948 ts_tablet_manager.cc:1397] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 06:46:16.105921 2950 tablet_bootstrap.cc:492] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: No bootstrap required, opened a new log
I20260809 06:46:16.106184 2950 ts_tablet_manager.cc:1397] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 06:46:16.111964 2945 raft_consensus.cc:348] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.112308 2945 raft_consensus.cc:374] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:16.112411 2945 raft_consensus.cc:729] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3843f75ce85d493797c776fc3d654dc6, State: Initialized, Role: FOLLOWER
I20260809 06:46:16.112965 2945 consensus_queue.cc:260] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.113997 2919 heartbeater.cc:503] Master 127.27.209.254:40937 was elected leader, sending a full tablet report...
I20260809 06:46:16.114825 2945 ts_tablet_manager.cc:1428] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260809 06:46:16.114959 2948 raft_consensus.cc:348] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.115439 2948 raft_consensus.cc:374] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:16.115581 2948 raft_consensus.cc:729] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370edeec11264dceb200fcba51f6a78b, State: Initialized, Role: FOLLOWER
I20260809 06:46:16.116182 2948 consensus_queue.cc:260] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.116459 2950 raft_consensus.cc:348] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.116955 2950 raft_consensus.cc:374] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:16.117096 2950 raft_consensus.cc:729] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0b3dbc984a4b9a917d23a1ed4550b7, State: Initialized, Role: FOLLOWER
I20260809 06:46:16.117734 2950 consensus_queue.cc:260] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.118355 2948 ts_tablet_manager.cc:1428] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260809 06:46:16.124302 2727 heartbeater.cc:503] Master 127.27.209.254:40937 was elected leader, sending a full tablet report...
I20260809 06:46:16.124724 2950 ts_tablet_manager.cc:1428] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Time spent starting tablet: real 0.018s user 0.003s sys 0.016s
I20260809 06:46:16.164588 2960 raft_consensus.cc:482] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:16.164880 2960 raft_consensus.cc:504] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.166569 2960 leader_election.cc:288] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 370edeec11264dceb200fcba51f6a78b (127.27.209.193:35973), 3843f75ce85d493797c776fc3d654dc6 (127.27.209.195:44873)
I20260809 06:46:16.175421 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.166998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:38927 (local address 127.27.209.193:35973)
0809 06:46:16.167217 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.167222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.167967 (+ 745us) server_negotiation.cc:407] Connection header received
0809 06:46:16.168168 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.168172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.168241 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.168340 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:16.169133 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.170215 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.173059 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.174238 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.174420 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.174496 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.175003 (+ 507us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.175173 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:46:16.175415 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.167005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:47443 (local address 127.27.209.195:44873)
0809 06:46:16.167205 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.167212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.167966 (+ 754us) server_negotiation.cc:407] Connection header received
0809 06:46:16.168144 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.168149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.168230 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.168335 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:16.169082 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.170216 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.173066 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.174276 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.174419 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.174496 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.175004 (+ 508us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.175148 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:46:16.175608 2961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.166980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:35973 (local address 127.27.209.194:38927)
0809 06:46:16.167855 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:16.167867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:16.168049 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:16.168350 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:16.168355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:16.168411 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:16.168988 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.168998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.170368 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.170374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.172894 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.172912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.174495 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.174501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.174772 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.174824 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 06:46:16.174919 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 06:46:16.175100 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":278,"spinlock_wait_cycles":68480,"thread_start_us":406,"threads_started":1}
I20260809 06:46:16.175969 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.166829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44873 (local address 127.27.209.194:47443)
0809 06:46:16.167826 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:16.167850 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:16.168007 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:16.168350 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:16.168355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:16.168414 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:16.168910 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.168929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.170368 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.170374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.172894 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:16.172912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.174499 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.174505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:16.174755 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.174820 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 06:46:16.174919 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 06:46:16.175100 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":167,"spinlock_wait_cycles":8064,"thread_start_us":636,"threads_started":1}
I20260809 06:46:16.176651 2491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b016ada4284411b6ec8c791f9b81d2" candidate_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370edeec11264dceb200fcba51f6a78b" is_pre_election: true
I20260809 06:46:16.176908 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b016ada4284411b6ec8c791f9b81d2" candidate_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3843f75ce85d493797c776fc3d654dc6" is_pre_election: true
I20260809 06:46:16.177307 2491 raft_consensus.cc:2454] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0b3dbc984a4b9a917d23a1ed4550b7 in term 0.
I20260809 06:46:16.177431 2850 raft_consensus.cc:2454] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0b3dbc984a4b9a917d23a1ed4550b7 in term 0.
I20260809 06:46:16.178112 2579 leader_election.cc:302] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370edeec11264dceb200fcba51f6a78b, 3b0b3dbc984a4b9a917d23a1ed4550b7; no voters:
I20260809 06:46:16.178577 2960 raft_consensus.cc:2781] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:16.178745 2960 raft_consensus.cc:482] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:16.178844 2960 raft_consensus.cc:3037] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:16.181555 2960 raft_consensus.cc:504] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.182101 2960 leader_election.cc:288] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [CANDIDATE]: Term 1 election: Requested vote from peers 370edeec11264dceb200fcba51f6a78b (127.27.209.193:35973), 3843f75ce85d493797c776fc3d654dc6 (127.27.209.195:44873)
I20260809 06:46:16.182572 2491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b016ada4284411b6ec8c791f9b81d2" candidate_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "370edeec11264dceb200fcba51f6a78b"
I20260809 06:46:16.182572 2850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b016ada4284411b6ec8c791f9b81d2" candidate_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3843f75ce85d493797c776fc3d654dc6"
I20260809 06:46:16.182799 2850 raft_consensus.cc:3037] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:16.182799 2491 raft_consensus.cc:3037] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:16.185976 2491 raft_consensus.cc:2454] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0b3dbc984a4b9a917d23a1ed4550b7 in term 1.
I20260809 06:46:16.185976 2850 raft_consensus.cc:2454] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0b3dbc984a4b9a917d23a1ed4550b7 in term 1.
I20260809 06:46:16.186396 2581 leader_election.cc:302] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3843f75ce85d493797c776fc3d654dc6, 3b0b3dbc984a4b9a917d23a1ed4550b7; no voters:
I20260809 06:46:16.186741 2960 raft_consensus.cc:2781] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:16.187230 2960 raft_consensus.cc:686] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 1 LEADER]: Becoming Leader. State: Replica: 3b0b3dbc984a4b9a917d23a1ed4550b7, State: Running, Role: LEADER
I20260809 06:46:16.187759 2960 consensus_queue.cc:237] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } }
I20260809 06:46:16.192528 2272 catalog_manager.cc:5495] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b0b3dbc984a4b9a917d23a1ed4550b7 (127.27.209.194). New cstate: current_term: 1 leader_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b0b3dbc984a4b9a917d23a1ed4550b7" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 35673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20260809 06:46:16.220674 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.215229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.27.209.194:35673)
0809 06:46:16.215465 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.215473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.215572 (+ 99us) server_negotiation.cc:407] Connection header received
0809 06:46:16.215780 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.215786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.215896 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.216021 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:16.216372 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.217180 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.218702 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.219126 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.219221 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.219643 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:16.220143 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:16.220256 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.220327 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.220400 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 06:46:16.221972 2552 tablet.cc:2369] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:16.228616 2491 raft_consensus.cc:1261] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 1 FOLLOWER]: Refusing update from remote peer 3b0b3dbc984a4b9a917d23a1ed4550b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:16.228617 2850 raft_consensus.cc:1261] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 1 FOLLOWER]: Refusing update from remote peer 3b0b3dbc984a4b9a917d23a1ed4550b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:16.229552 2963 consensus_queue.cc:1035] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3843f75ce85d493797c776fc3d654dc6" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:16.229895 2960 consensus_queue.cc:1035] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "370edeec11264dceb200fcba51f6a78b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:46:16.237742 2920 tablet.cc:2369] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:16.238618 2967 mvcc.cc:205] Tried to move back new op lower bound from 7316512670623084544 to 7316512670468939776. Current Snapshot: MvccSnapshot[applied={T|T < 7316512670623084544}]
I20260809 06:46:16.242358 2968 mvcc.cc:205] Tried to move back new op lower bound from 7316512670623084544 to 7316512670468939776. Current Snapshot: MvccSnapshot[applied={T|T < 7316512670623084544}]
I20260809 06:46:16.250725 2969 mvcc.cc:205] Tried to move back new op lower bound from 7316512670623084544 to 7316512670468939776. Current Snapshot: MvccSnapshot[applied={T|T < 7316512670623084544}]
I20260809 06:46:16.253634 2273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:16.253989 2273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:16.258348 2273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07ed8785d45645e79c37500671a9cfd9: Sending DeleteTablet for 3 replicas of tablet e7b016ada4284411b6ec8c791f9b81d2
I20260809 06:46:16.259584 2458 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b016ada4284411b6ec8c791f9b81d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:16 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:57478
I20260809 06:46:16.263036 2639 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b016ada4284411b6ec8c791f9b81d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:16 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:49298
I20260809 06:46:16.264086 2824 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b016ada4284411b6ec8c791f9b81d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:16 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:60032
I20260809 06:46:16.266666 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 2366
I20260809 06:46:16.272032 2983 tablet_replica.cc:330] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: stopping tablet replica
I20260809 06:46:16.272032 2980 tablet_replica.cc:330] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: stopping tablet replica
I20260809 06:46:16.275625 2980 raft_consensus.cc:2229] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:16.276219 2980 raft_consensus.cc:2258] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:16.276806 2983 raft_consensus.cc:2229] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:16.277226 2982 tablet_replica.cc:330] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: stopping tablet replica
I20260809 06:46:16.278409 2983 raft_consensus.cc:2258] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:16.278888 2982 raft_consensus.cc:2229] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:16.279731 2980 ts_tablet_manager.cc:1905] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:16.283197 2983 ts_tablet_manager.cc:1905] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:16.284250 2982 raft_consensus.cc:2258] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:16.286047 2982 ts_tablet_manager.cc:1905] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:46:16 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:16.287590 2980 ts_tablet_manager.cc:1918] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:16.287791 2980 log.cc:1196] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/e7b016ada4284411b6ec8c791f9b81d2
I20260809 06:46:16.288367 2980 ts_tablet_manager.cc:1939] T e7b016ada4284411b6ec8c791f9b81d2 P 370edeec11264dceb200fcba51f6a78b: Deleting consensus metadata
I20260809 06:46:16.290310 2253 catalog_manager.cc:4841] TS 370edeec11264dceb200fcba51f6a78b (127.27.209.193:35973): tablet e7b016ada4284411b6ec8c791f9b81d2 (table test-table [id=68324dcc60f8492baf539215b580db4d]) successfully deleted
I20260809 06:46:16.294946 2982 ts_tablet_manager.cc:1918] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:16.294976 2983 ts_tablet_manager.cc:1918] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:16.295113 2982 log.cc:1196] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/e7b016ada4284411b6ec8c791f9b81d2
I20260809 06:46:16.295146 2983 log.cc:1196] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/e7b016ada4284411b6ec8c791f9b81d2
I20260809 06:46:16.295475 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.267957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.27.209.193:35973)
0809 06:46:16.268147 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.268154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.272966 (+ 4812us) server_negotiation.cc:407] Connection header received
0809 06:46:16.273238 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.273244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.273330 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.273405 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:16.278816 (+ 5411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.279864 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.281721 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.282473 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.282603 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.287918 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:16.287932 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:16.287950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:16.287998 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:16.291923 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.292811 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.292821 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.292825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.292991 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.293481 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.293486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.293489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.293969 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:16.294276 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.294859 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.295098 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:46:16.295696 2983 ts_tablet_manager.cc:1939] T e7b016ada4284411b6ec8c791f9b81d2 P 3843f75ce85d493797c776fc3d654dc6: Deleting consensus metadata
I20260809 06:46:16.295780 2982 ts_tablet_manager.cc:1939] T e7b016ada4284411b6ec8c791f9b81d2 P 3b0b3dbc984a4b9a917d23a1ed4550b7: Deleting consensus metadata
I20260809 06:46:16.296904 2543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:16.297436 2255 catalog_manager.cc:4841] TS 3843f75ce85d493797c776fc3d654dc6 (127.27.209.195:44873): tablet e7b016ada4284411b6ec8c791f9b81d2 (table test-table [id=68324dcc60f8492baf539215b580db4d]) successfully deleted
I20260809 06:46:16.297673 2253 catalog_manager.cc:4841] TS 3b0b3dbc984a4b9a917d23a1ed4550b7 (127.27.209.194:35673): tablet e7b016ada4284411b6ec8c791f9b81d2 (table test-table [id=68324dcc60f8492baf539215b580db4d]) successfully deleted
I20260809 06:46:16.385636 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 2366
I20260809 06:46:16.407059 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 2560
Aug 09 06:46:16 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:46:16.422878 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.408083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.27.209.194:35673)
0809 06:46:16.408293 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.408298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.408314 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:46:16.408373 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.408376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.408440 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.408525 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:16.409822 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.410698 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.412061 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.412514 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.412634 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.416917 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:16.416932 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:16.416949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:16.416988 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:16.420109 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.420737 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.420745 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.420747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.420924 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.421298 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.421303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.421306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.421668 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:16.421997 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.422303 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.422518 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:46:16.423663 2721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:16.543466 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 2560
I20260809 06:46:16.571141 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 2734
Aug 09 06:46:16 dist-test-slave-jpj1 krb5kdc[2207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:46:16.585526 2941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:16.572009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.27.209.195:44873)
0809 06:46:16.572174 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:16.572179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:16.572197 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:46:16.572256 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:16.572259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:16.572322 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:16.572410 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:16.573599 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.574320 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.575514 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:16.575983 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:16.576107 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:16.580349 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:16.580363 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:16.580380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:16.580420 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:16.583045 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.583781 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.583789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.583791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.583942 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:16.584246 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:16.584249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:16.584251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:16.584609 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:16.584830 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:16.585038 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 06:46:16.585218 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 06:46:16.586179 2912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:16.673427 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 2734
I20260809 06:46:16.695104 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 2227
I20260809 06:46:16.696305 2312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:16.802343 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 2227
2026-08-09T06:46:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4236 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T06:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:16Z Disabled control of system clock
I20260809 06:46:16.874047 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41379
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:41379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 06:46:17.197381 2998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:17.197666 2998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:17.197801 2998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:17.205026 2998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:17.205143 2998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:17.205211 2998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:17.205296 2998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:17.205356 2998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:17.214638 2998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:41379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:17.215998 2998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:17.217876 2998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:17.229686 3006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:17.229764 3007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:17.229743 3009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:17.230101 2998 server_base.cc:1034] running on GCE node
I20260809 06:46:17.231135 2998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:17.232529 2998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:17.233779 2998 hybrid_clock.cc:648] HybridClock initialized: now 1786257977233729 us; error 28 us; skew 500 ppm
I20260809 06:46:17.234184 2998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:17.236546 2998 webserver.cc:460] Webserver started at http://127.27.209.254:46559/ using document root <none> and password file <none>
I20260809 06:46:17.237268 2998 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:17.237371 2998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:17.237711 2998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:17.240787 2998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "cbed1744b65b4e76b339e0bae38a6aad"
format_stamp: "Formatted at 2026-08-09 06:46:17 on dist-test-slave-jpj1"
server_key: "49b911f9b4d7ede58259e59650214da0"
server_key_iv: "14b9224581ce619c610a6b3f35e38130"
server_key_version: "encryptionkey@0"
I20260809 06:46:17.241597 2998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "cbed1744b65b4e76b339e0bae38a6aad"
format_stamp: "Formatted at 2026-08-09 06:46:17 on dist-test-slave-jpj1"
server_key: "49b911f9b4d7ede58259e59650214da0"
server_key_iv: "14b9224581ce619c610a6b3f35e38130"
server_key_version: "encryptionkey@0"
I20260809 06:46:17.246850 2998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260809 06:46:17.250483 3020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:17.251861 2998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:17.252023 2998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "cbed1744b65b4e76b339e0bae38a6aad"
format_stamp: "Formatted at 2026-08-09 06:46:17 on dist-test-slave-jpj1"
server_key: "49b911f9b4d7ede58259e59650214da0"
server_key_iv: "14b9224581ce619c610a6b3f35e38130"
server_key_version: "encryptionkey@0"
I20260809 06:46:17.252231 2998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:17.257869 2998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:17.258625 2998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:17.258903 2998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:17.276206 2998 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:41379
I20260809 06:46:17.276247 3090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:41379 every 8 connection(s)
I20260809 06:46:17.277596 2998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:17.281155 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 2998
I20260809 06:46:17.281356 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:17.281450 3091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:17.282012 28487 external_mini_cluster.cc:1428] Setting key 63933bd39efdc7cfa873cfbc7a0b678a
I20260809 06:46:17.291961 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:17.284014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.27.209.254:41379)
0809 06:46:17.285149 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:17.285163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:17.285223 (+ 60us) server_negotiation.cc:407] Connection header received
0809 06:46:17.287882 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:17.287913 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:17.288484 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:17.288954 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:17.289950 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:17.289958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:17.289975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:17.289991 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:17.290067 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:17.290087 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:17.290303 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:17.290443 (+ 140us) server_negotiation.cc:299] Negotiation successful
0809 06:46:17.290732 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":362,"threads_started":1}
I20260809 06:46:17.295979 3091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad: Bootstrap starting.
I20260809 06:46:17.298834 3091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:17.299973 3091 log.cc:824] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:17.302809 3091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad: No bootstrap required, opened a new log
I20260809 06:46:17.310297 3091 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbed1744b65b4e76b339e0bae38a6aad" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41379 } }
I20260809 06:46:17.310635 3091 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:17.310732 3091 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbed1744b65b4e76b339e0bae38a6aad, State: Initialized, Role: FOLLOWER
I20260809 06:46:17.311283 3091 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbed1744b65b4e76b339e0bae38a6aad" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41379 } }
I20260809 06:46:17.311491 3091 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:17.311637 3091 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:17.311784 3091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:17.313855 3091 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbed1744b65b4e76b339e0bae38a6aad" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41379 } }
I20260809 06:46:17.314339 3091 leader_election.cc:302] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbed1744b65b4e76b339e0bae38a6aad; no voters:
I20260809 06:46:17.314914 3091 leader_election.cc:288] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:17.315035 3097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:17.315833 3097 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [term 1 LEADER]: Becoming Leader. State: Replica: cbed1744b65b4e76b339e0bae38a6aad, State: Running, Role: LEADER
I20260809 06:46:17.316473 3097 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbed1744b65b4e76b339e0bae38a6aad" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41379 } }
I20260809 06:46:17.317090 3091 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:17.321326 3100 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbed1744b65b4e76b339e0bae38a6aad. Latest consensus state: current_term: 1 leader_uuid: "cbed1744b65b4e76b339e0bae38a6aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbed1744b65b4e76b339e0bae38a6aad" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41379 } } }
I20260809 06:46:17.321321 3099 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbed1744b65b4e76b339e0bae38a6aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbed1744b65b4e76b339e0bae38a6aad" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41379 } } }
I20260809 06:46:17.321777 3100 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:17.321815 3099 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:17.323485 3113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:17.328486 3113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:17.336809 3113 catalog_manager.cc:1347] Generated new cluster ID: 78530f261f52451bbf046b8b73f96785
I20260809 06:46:17.336938 3113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:17.354755 3113 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:17.355285 3113 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:17.369887 3113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad: Generated new TSK 0
I20260809 06:46:17.370941 3113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:46:17.380823 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:41379
--builtin_ntp_servers=127.27.209.212:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 06:46:17.716744 3129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:17.717133 3129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:17.717257 3129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:17.724519 3129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:17.724660 3129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:17.724810 3129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:17.733662 3129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:41379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:17.734917 3129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:17.736788 3129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:17.749344 3138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:17.749382 3140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:17.749347 3137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:17.750176 3129 server_base.cc:1034] running on GCE node
I20260809 06:46:17.750804 3129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:17.751783 3129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:17.752993 3129 hybrid_clock.cc:648] HybridClock initialized: now 1786257977752937 us; error 50 us; skew 500 ppm
I20260809 06:46:17.753312 3129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:17.755563 3129 webserver.cc:460] Webserver started at http://127.27.209.193:38479/ using document root <none> and password file <none>
I20260809 06:46:17.756299 3129 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:17.756422 3129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:17.756762 3129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:17.759604 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "a2424aa81fbe49e7b870c4862f41f113"
format_stamp: "Formatted at 2026-08-09 06:46:17 on dist-test-slave-jpj1"
server_key: "203fcdd7582d0946bdaea6ce00277caf"
server_key_iv: "dd3e9cd26aaf1fa978fee6aa6bd35b9f"
server_key_version: "encryptionkey@0"
I20260809 06:46:17.760398 3129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "a2424aa81fbe49e7b870c4862f41f113"
format_stamp: "Formatted at 2026-08-09 06:46:17 on dist-test-slave-jpj1"
server_key: "203fcdd7582d0946bdaea6ce00277caf"
server_key_iv: "dd3e9cd26aaf1fa978fee6aa6bd35b9f"
server_key_version: "encryptionkey@0"
I20260809 06:46:17.765573 3129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:46:17.768996 3146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:17.770324 3129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:46:17.770470 3129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "a2424aa81fbe49e7b870c4862f41f113"
format_stamp: "Formatted at 2026-08-09 06:46:17 on dist-test-slave-jpj1"
server_key: "203fcdd7582d0946bdaea6ce00277caf"
server_key_iv: "dd3e9cd26aaf1fa978fee6aa6bd35b9f"
server_key_version: "encryptionkey@0"
I20260809 06:46:17.770670 3129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:17.776149 3129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:17.776889 3129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:17.777127 3129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:17.777998 3129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:17.779326 3129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:17.779419 3129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:17.779546 3129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:17.779629 3129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 06:46:17.809844 3129 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:45179
I20260809 06:46:17.809897 3310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:45179 every 8 connection(s)
I20260809 06:46:17.811143 3129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:17.812203 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 3129
I20260809 06:46:17.812419 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:17.813246 28487 external_mini_cluster.cc:1428] Setting key 0a15e7fd7207236c97848ce42a0d5685
I20260809 06:46:17.816450 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:41379
--builtin_ntp_servers=127.27.209.212:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 06:46:17.820461 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:17.813944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:42893 (local address 127.27.209.254:41379)
0809 06:46:17.814386 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:17.814390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:17.815862 (+ 1472us) server_negotiation.cc:407] Connection header received
0809 06:46:17.818239 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:17.818243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:17.818294 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:17.818395 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:17.819815 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:17.819820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:17.819823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:17.819839 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:17.819883 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:17.819894 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:17.819985 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:17.820168 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 06:46:17.820214 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260809 06:46:17.821637 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:17.814561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:41379 (local address 127.27.209.193:42893)
0809 06:46:17.815695 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:17.815744 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:17.817957 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:17.818645 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:17.818655 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:17.818771 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:17.819469 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:17.819472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:17.819522 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:17.819652 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:17.819981 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:17.819991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:17.819997 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 06:46:17.820255 (+ 258us) client_negotiation.cc:240] Negotiation successful
0809 06:46:17.820507 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":315,"threads_started":1}
I20260809 06:46:17.823046 3311 heartbeater.cc:347] Connected to a master server at 127.27.209.254:41379
I20260809 06:46:17.823345 3311 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:17.824029 3311 heartbeater.cc:511] Master 127.27.209.254:41379 requested a full tablet report, sending...
I20260809 06:46:17.826294 3043 ts_manager.cc:194] Registered new tserver with Master: a2424aa81fbe49e7b870c4862f41f113 (127.27.209.193:45179)
W20260809 06:46:18.141836 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:18.142108 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:18.142196 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:18.149367 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:18.149468 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:18.149575 3315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:18.159032 3315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:41379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:18.160360 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:18.162223 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:18.175559 3322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:18.175642 3323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:18.175675 3328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:18.176695 3315 server_base.cc:1034] running on GCE node
I20260809 06:46:18.177350 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:18.178376 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:18.179654 3315 hybrid_clock.cc:648] HybridClock initialized: now 1786257978179554 us; error 55 us; skew 500 ppm
I20260809 06:46:18.179983 3315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:18.182288 3315 webserver.cc:460] Webserver started at http://127.27.209.194:37337/ using document root <none> and password file <none>
I20260809 06:46:18.182994 3315 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:18.183099 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:18.183436 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:18.187196 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "af03f3867e294ebab0e2e3a3f274ca1c"
format_stamp: "Formatted at 2026-08-09 06:46:18 on dist-test-slave-jpj1"
server_key: "29a37e70914b792b3174650953c7266c"
server_key_iv: "b4e202c10776b4decebf758bd9829e6d"
server_key_version: "encryptionkey@0"
I20260809 06:46:18.188032 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "af03f3867e294ebab0e2e3a3f274ca1c"
format_stamp: "Formatted at 2026-08-09 06:46:18 on dist-test-slave-jpj1"
server_key: "29a37e70914b792b3174650953c7266c"
server_key_iv: "b4e202c10776b4decebf758bd9829e6d"
server_key_version: "encryptionkey@0"
I20260809 06:46:18.193236 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:46:18.196569 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:18.197842 3315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:18.197994 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "af03f3867e294ebab0e2e3a3f274ca1c"
format_stamp: "Formatted at 2026-08-09 06:46:18 on dist-test-slave-jpj1"
server_key: "29a37e70914b792b3174650953c7266c"
server_key_iv: "b4e202c10776b4decebf758bd9829e6d"
server_key_version: "encryptionkey@0"
I20260809 06:46:18.198196 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:18.203753 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:18.204504 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:18.204752 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:18.205622 3315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:18.206948 3315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:18.207046 3315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:18.207158 3315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:18.207228 3315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:18.237540 3315 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:37953
I20260809 06:46:18.237588 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:37953 every 8 connection(s)
I20260809 06:46:18.238834 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:18.239488 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 3315
I20260809 06:46:18.239699 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:18.240237 28487 external_mini_cluster.cc:1428] Setting key 0389545abb6153011b5e4f2379ed0c46
I20260809 06:46:18.243350 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:41379
--builtin_ntp_servers=127.27.209.212:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 06:46:18.247985 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.241469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:45541 (local address 127.27.209.254:41379)
0809 06:46:18.241676 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:18.241683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:18.243347 (+ 1664us) server_negotiation.cc:407] Connection header received
0809 06:46:18.245690 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:18.245694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:18.245747 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:18.245838 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:18.247285 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:18.247290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:18.247295 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:18.247313 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:18.247349 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:18.247362 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:18.247459 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:18.247666 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 06:46:18.247721 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:46:18.249158 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.242068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:41379 (local address 127.27.209.194:45541)
0809 06:46:18.243159 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:18.243220 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:18.245348 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:18.246091 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:18.246102 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:18.246216 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:18.246941 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:18.246944 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:18.246993 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:18.247101 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:18.247447 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:18.247456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:18.247464 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:46:18.247752 (+ 288us) client_negotiation.cc:240] Negotiation successful
0809 06:46:18.248015 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":291,"threads_started":1}
I20260809 06:46:18.250483 3495 heartbeater.cc:347] Connected to a master server at 127.27.209.254:41379
I20260809 06:46:18.250804 3495 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:18.251483 3495 heartbeater.cc:511] Master 127.27.209.254:41379 requested a full tablet report, sending...
I20260809 06:46:18.252985 3043 ts_manager.cc:194] Registered new tserver with Master: af03f3867e294ebab0e2e3a3f274ca1c (127.27.209.194:37953)
W20260809 06:46:18.575217 3499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:18.575470 3499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:18.575618 3499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:18.582588 3499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:18.582700 3499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:18.582854 3499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:18.591655 3499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:41379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:18.592906 3499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:18.594791 3499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:18.607367 3506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:18.607326 3509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:18.607391 3507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:18.608100 3499 server_base.cc:1034] running on GCE node
I20260809 06:46:18.608671 3499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:18.609539 3499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:18.610745 3499 hybrid_clock.cc:648] HybridClock initialized: now 1786257978610695 us; error 41 us; skew 500 ppm
I20260809 06:46:18.611045 3499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:18.613174 3499 webserver.cc:460] Webserver started at http://127.27.209.195:45501/ using document root <none> and password file <none>
I20260809 06:46:18.613834 3499 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:18.613929 3499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:18.614243 3499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:18.617053 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "197d13cf275e492293b90dba861ad0ac"
format_stamp: "Formatted at 2026-08-09 06:46:18 on dist-test-slave-jpj1"
server_key: "0dc7f19f04694aaa5e8d3571fec6dbbc"
server_key_iv: "cc28bbc6e08e825d86671c12c1de604d"
server_key_version: "encryptionkey@0"
I20260809 06:46:18.617849 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "197d13cf275e492293b90dba861ad0ac"
format_stamp: "Formatted at 2026-08-09 06:46:18 on dist-test-slave-jpj1"
server_key: "0dc7f19f04694aaa5e8d3571fec6dbbc"
server_key_iv: "cc28bbc6e08e825d86671c12c1de604d"
server_key_version: "encryptionkey@0"
I20260809 06:46:18.622901 3499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:46:18.626258 3517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:18.627612 3499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:18.627785 3499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "197d13cf275e492293b90dba861ad0ac"
format_stamp: "Formatted at 2026-08-09 06:46:18 on dist-test-slave-jpj1"
server_key: "0dc7f19f04694aaa5e8d3571fec6dbbc"
server_key_iv: "cc28bbc6e08e825d86671c12c1de604d"
server_key_version: "encryptionkey@0"
I20260809 06:46:18.627980 3499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:18.633368 3499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:18.634100 3499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:18.634337 3499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:18.635201 3499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:18.636577 3499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:18.636678 3499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:18.636785 3499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:18.636852 3499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:18.666453 3499 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:44533
I20260809 06:46:18.666510 3669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:44533 every 8 connection(s)
I20260809 06:46:18.667757 3499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:18.675678 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.670160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:34975 (local address 127.27.209.254:41379)
0809 06:46:18.670331 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:18.670336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:18.671779 (+ 1443us) server_negotiation.cc:407] Connection header received
0809 06:46:18.673433 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:18.673439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:18.673503 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:18.673592 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:18.674928 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:18.674935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:18.674940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:18.674960 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:18.674995 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:18.675006 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:18.675120 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:18.675303 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 06:46:18.675371 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:46:18.675936 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 3499
I20260809 06:46:18.676091 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:18.676520 28487 external_mini_cluster.cc:1428] Setting key 27eddbb52e43608074a71f5bd4ecf196
I20260809 06:46:18.676626 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.670664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:41379 (local address 127.27.209.195:34975)
0809 06:46:18.671610 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:18.671655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:18.673168 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:18.673875 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:18.673883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:18.673977 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:18.674602 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:18.674604 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:18.674642 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:18.674735 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:18.675099 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:18.675110 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:18.675117 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:46:18.675372 (+ 255us) client_negotiation.cc:240] Negotiation successful
0809 06:46:18.675662 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":276,"threads_started":1}
I20260809 06:46:18.677977 3670 heartbeater.cc:347] Connected to a master server at 127.27.209.254:41379
I20260809 06:46:18.678256 3670 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:18.678864 3670 heartbeater.cc:511] Master 127.27.209.254:41379 requested a full tablet report, sending...
I20260809 06:46:18.680259 3041 ts_manager.cc:194] Registered new tserver with Master: 197d13cf275e492293b90dba861ad0ac (127.27.209.195:44533)
I20260809 06:46:18.691303 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:18.698087 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.695732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.27.209.254:41379)
0809 06:46:18.695920 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:18.695927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:18.696346 (+ 419us) server_negotiation.cc:407] Connection header received
0809 06:46:18.696506 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:18.696509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:18.696559 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:18.696636 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:18.697493 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:18.697498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:18.697503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:18.697519 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:18.697550 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:18.697561 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:18.697662 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:18.697766 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 06:46:18.697826 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:46:18.701762 3041 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:18.708330 3041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:18.736649 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.719900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36460)
0809 06:46:18.732684 (+ 12784us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:18.732712 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:18.732805 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:18.733892 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:18.733900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:18.734000 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:18.734900 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:18.734904 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:18.734937 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:18.735031 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:18.735583 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:18.735598 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:18.735606 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:46:18.735778 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 06:46:18.735865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12632,"thread_start_us":12973,"threads_started":1}
I20260809 06:46:18.737000 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.720135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.27.209.193:45179)
0809 06:46:18.733430 (+ 13295us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:18.733437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:18.733462 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:46:18.733518 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:18.733524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:18.733716 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:18.733857 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:18.735245 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:18.735254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:18.735270 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:18.735289 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:18.735351 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:18.735372 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:18.735501 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:18.736503 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0809 06:46:18.736612 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13152,"thread_start_us":13035,"threads_started":1}
I20260809 06:46:18.737870 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.722187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.27.209.194:37953)
0809 06:46:18.730062 (+ 7875us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:18.730073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:18.734679 (+ 4606us) server_negotiation.cc:407] Connection header received
0809 06:46:18.735126 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:18.735135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:18.735358 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:18.735477 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:18.736962 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:18.736971 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:18.736985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:18.737015 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:18.737074 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:18.737096 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:18.737257 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:18.737508 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 06:46:18.737604 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7673,"thread_start_us":7524,"threads_started":1}
I20260809 06:46:18.737890 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.721515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:37953 (local address 127.0.0.1:40168)
0809 06:46:18.734549 (+ 13034us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:18.734567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:18.734950 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:18.735969 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:18.735974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:18.735997 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:18.736715 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:18.736719 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:18.736735 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:18.736779 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:18.737318 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:18.737323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:18.737328 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:46:18.737467 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 06:46:18.737517 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12911,"thread_start_us":9595,"threads_started":1}
I20260809 06:46:18.739292 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.719982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.27.209.195:44533)
0809 06:46:18.735202 (+ 15220us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:18.735210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:18.735242 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:46:18.735304 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:18.735311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:18.735515 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:18.735679 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:18.738455 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:18.738464 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:18.738477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:18.738496 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:18.738553 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:18.738572 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:18.738760 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:18.738822 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 06:46:18.738915 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15047,"thread_start_us":15671,"threads_started":1}
I20260809 06:46:18.739303 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:18.719624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44533 (local address 127.0.0.1:45882)
0809 06:46:18.732130 (+ 12506us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:18.732182 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:18.732344 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:18.736707 (+ 4363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:18.736711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:18.736985 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:18.738200 (+ 1215us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:18.738204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:18.738221 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:18.738266 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:18.738693 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:18.738699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:18.738704 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:46:18.738833 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 06:46:18.738882 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11636,"thread_start_us":15080,"threads_started":1}
I20260809 06:46:18.741370 3208 tablet_service.cc:1467] Processing CreateTablet for tablet dcbcff934c87484e8fb2861e185df4c1 (DEFAULT_TABLE table=test-table [id=88fe3b7c911c4dfa95ae3f2d210febb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:18.741428 3400 tablet_service.cc:1467] Processing CreateTablet for tablet dcbcff934c87484e8fb2861e185df4c1 (DEFAULT_TABLE table=test-table [id=88fe3b7c911c4dfa95ae3f2d210febb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:18.742692 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbcff934c87484e8fb2861e185df4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:18.742865 3400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbcff934c87484e8fb2861e185df4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:18.743484 3583 tablet_service.cc:1467] Processing CreateTablet for tablet dcbcff934c87484e8fb2861e185df4c1 (DEFAULT_TABLE table=test-table [id=88fe3b7c911c4dfa95ae3f2d210febb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:18.745369 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbcff934c87484e8fb2861e185df4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:18.753877 3697 tablet_bootstrap.cc:492] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: Bootstrap starting.
I20260809 06:46:18.756951 3698 tablet_bootstrap.cc:492] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Bootstrap starting.
I20260809 06:46:18.758433 3697 tablet_bootstrap.cc:654] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:18.759229 3699 tablet_bootstrap.cc:492] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Bootstrap starting.
I20260809 06:46:18.759737 3697 log.cc:824] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:18.760192 3698 tablet_bootstrap.cc:654] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:18.761209 3698 log.cc:824] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:18.762533 3699 tablet_bootstrap.cc:654] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:18.763113 3697 tablet_bootstrap.cc:492] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: No bootstrap required, opened a new log
I20260809 06:46:18.763360 3697 ts_tablet_manager.cc:1397] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 06:46:18.763713 3699 log.cc:824] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:18.764160 3698 tablet_bootstrap.cc:492] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: No bootstrap required, opened a new log
I20260809 06:46:18.764446 3698 ts_tablet_manager.cc:1397] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 06:46:18.766638 3699 tablet_bootstrap.cc:492] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: No bootstrap required, opened a new log
I20260809 06:46:18.766892 3699 ts_tablet_manager.cc:1397] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 06:46:18.770610 3697 raft_consensus.cc:348] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:18.771071 3697 raft_consensus.cc:374] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:18.771173 3697 raft_consensus.cc:729] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2424aa81fbe49e7b870c4862f41f113, State: Initialized, Role: FOLLOWER
I20260809 06:46:18.771816 3697 consensus_queue.cc:260] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:18.772836 3311 heartbeater.cc:503] Master 127.27.209.254:41379 was elected leader, sending a full tablet report...
I20260809 06:46:18.773350 3697 ts_tablet_manager.cc:1428] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260809 06:46:18.774549 3698 raft_consensus.cc:348] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:18.774982 3698 raft_consensus.cc:374] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:18.775087 3698 raft_consensus.cc:729] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af03f3867e294ebab0e2e3a3f274ca1c, State: Initialized, Role: FOLLOWER
I20260809 06:46:18.775705 3698 consensus_queue.cc:260] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:18.776782 3495 heartbeater.cc:503] Master 127.27.209.254:41379 was elected leader, sending a full tablet report...
I20260809 06:46:18.776719 3699 raft_consensus.cc:348] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:18.777117 3699 raft_consensus.cc:374] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:18.777223 3699 raft_consensus.cc:729] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197d13cf275e492293b90dba861ad0ac, State: Initialized, Role: FOLLOWER
I20260809 06:46:18.777397 3698 ts_tablet_manager.cc:1428] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 06:46:18.777763 3699 consensus_queue.cc:260] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:18.778820 3670 heartbeater.cc:503] Master 127.27.209.254:41379 was elected leader, sending a full tablet report...
I20260809 06:46:18.779393 3699 ts_tablet_manager.cc:1428] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260809 06:46:18.813341 3312 tablet.cc:2369] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:46:18.919399 3671 tablet.cc:2369] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:46:18.990985 3496 tablet.cc:2369] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:19.069630 3708 raft_consensus.cc:482] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:19.069962 3708 raft_consensus.cc:504] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:19.071715 3708 leader_election.cc:288] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197d13cf275e492293b90dba861ad0ac (127.27.209.195:44533), a2424aa81fbe49e7b870c4862f41f113 (127.27.209.193:45179)
I20260809 06:46:19.075093 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.072215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:37569 (local address 127.27.209.193:45179)
0809 06:46:19.072409 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:19.072416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:19.073031 (+ 615us) server_negotiation.cc:407] Connection header received
0809 06:46:19.073161 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:19.073167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:19.073233 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:19.073330 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:19.074465 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:19.074471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:19.074476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:19.074496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:19.074533 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:19.074547 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:19.074664 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:19.074756 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 06:46:19.074818 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:46:19.075227 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.072104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:53151 (local address 127.27.209.195:44533)
0809 06:46:19.072304 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:19.072311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:19.073175 (+ 864us) server_negotiation.cc:407] Connection header received
0809 06:46:19.073287 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:19.073292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:19.073355 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:19.073446 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:19.074547 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:19.074551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:19.074554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:19.074569 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:19.074617 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:19.074633 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:19.074758 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:19.074895 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 06:46:19.074972 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:46:19.075326 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.072004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.27.209.194:37569)
0809 06:46:19.072909 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.072925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:19.073040 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:19.073364 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:19.073369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:19.073392 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:19.074244 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:19.074248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:19.074270 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:19.074334 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:19.074645 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:19.074650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:19.074655 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:46:19.074823 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 06:46:19.074873 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":527,"threads_started":1}
I20260809 06:46:19.075456 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.072013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44533 (local address 127.27.209.194:53151)
0809 06:46:19.073040 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.073058 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:19.073174 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:19.073477 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:19.073480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:19.073498 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:19.074319 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:46:19.074322 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:19.074339 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:46:19.074383 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:19.074726 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:19.074731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:19.074735 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 06:46:19.074938 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 06:46:19.074983 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":1,"thread_start_us":607,"threads_started":1}
I20260809 06:46:19.076408 3604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbcff934c87484e8fb2861e185df4c1" candidate_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197d13cf275e492293b90dba861ad0ac" is_pre_election: true
I20260809 06:46:19.076409 3243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbcff934c87484e8fb2861e185df4c1" candidate_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2424aa81fbe49e7b870c4862f41f113" is_pre_election: true
I20260809 06:46:19.076999 3604 raft_consensus.cc:2454] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af03f3867e294ebab0e2e3a3f274ca1c in term 0.
I20260809 06:46:19.076999 3243 raft_consensus.cc:2454] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af03f3867e294ebab0e2e3a3f274ca1c in term 0.
I20260809 06:46:19.077711 3336 leader_election.cc:302] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2424aa81fbe49e7b870c4862f41f113, af03f3867e294ebab0e2e3a3f274ca1c; no voters:
I20260809 06:46:19.078155 3708 raft_consensus.cc:2781] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:19.078310 3708 raft_consensus.cc:482] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:19.078409 3708 raft_consensus.cc:3037] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:19.080750 3708 raft_consensus.cc:504] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:19.081396 3708 leader_election.cc:288] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [CANDIDATE]: Term 1 election: Requested vote from peers 197d13cf275e492293b90dba861ad0ac (127.27.209.195:44533), a2424aa81fbe49e7b870c4862f41f113 (127.27.209.193:45179)
I20260809 06:46:19.081818 3604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbcff934c87484e8fb2861e185df4c1" candidate_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197d13cf275e492293b90dba861ad0ac"
I20260809 06:46:19.081894 3243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbcff934c87484e8fb2861e185df4c1" candidate_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2424aa81fbe49e7b870c4862f41f113"
I20260809 06:46:19.082005 3604 raft_consensus.cc:3037] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:19.082089 3243 raft_consensus.cc:3037] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:19.084255 3604 raft_consensus.cc:2454] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af03f3867e294ebab0e2e3a3f274ca1c in term 1.
I20260809 06:46:19.084326 3243 raft_consensus.cc:2454] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af03f3867e294ebab0e2e3a3f274ca1c in term 1.
I20260809 06:46:19.084676 3339 leader_election.cc:302] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197d13cf275e492293b90dba861ad0ac, af03f3867e294ebab0e2e3a3f274ca1c; no voters:
I20260809 06:46:19.084975 3708 raft_consensus.cc:2781] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:19.085397 3708 raft_consensus.cc:686] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 1 LEADER]: Becoming Leader. State: Replica: af03f3867e294ebab0e2e3a3f274ca1c, State: Running, Role: LEADER
I20260809 06:46:19.085884 3708 consensus_queue.cc:237] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } }
I20260809 06:46:19.090782 3041 catalog_manager.cc:5495] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c reported cstate change: term changed from 0 to 1, leader changed from <none> to af03f3867e294ebab0e2e3a3f274ca1c (127.27.209.194). New cstate: current_term: 1 leader_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af03f3867e294ebab0e2e3a3f274ca1c" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37953 } health_report { overall_health: HEALTHY } } }
I20260809 06:46:19.102525 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.100712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.27.209.194:37953)
0809 06:46:19.100857 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:19.100862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:19.100993 (+ 131us) server_negotiation.cc:407] Connection header received
0809 06:46:19.101153 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:19.101156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:19.101234 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:19.101315 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:19.102096 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:19.102100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:19.102103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:19.102119 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:19.102150 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:19.102162 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:19.102262 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:19.102300 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 06:46:19.102343 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:46:19.110360 3604 raft_consensus.cc:1261] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 1 FOLLOWER]: Refusing update from remote peer af03f3867e294ebab0e2e3a3f274ca1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:19.110358 3243 raft_consensus.cc:1261] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 1 FOLLOWER]: Refusing update from remote peer af03f3867e294ebab0e2e3a3f274ca1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:19.111322 3713 consensus_queue.cc:1035] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "197d13cf275e492293b90dba861ad0ac" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:19.111673 3708 consensus_queue.cc:1035] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2424aa81fbe49e7b870c4862f41f113" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:19.136031 3043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:19.136317 3043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:19.140053 3043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbed1744b65b4e76b339e0bae38a6aad: Sending DeleteTablet for 3 replicas of tablet dcbcff934c87484e8fb2861e185df4c1
I20260809 06:46:19.141151 3583 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbcff934c87484e8fb2861e185df4c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:19 UTC) from {username='slave'} at 127.0.0.1:45882
I20260809 06:46:19.141179 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbcff934c87484e8fb2861e185df4c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:19 UTC) from {username='slave'} at 127.0.0.1:36460
I20260809 06:46:19.141397 3400 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbcff934c87484e8fb2861e185df4c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:19 UTC) from {username='slave'} at 127.0.0.1:40168
I20260809 06:46:19.143960 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 3129
I20260809 06:46:19.152123 3736 tablet_replica.cc:330] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: stopping tablet replica
I20260809 06:46:19.153254 3736 raft_consensus.cc:2229] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:19.154626 3736 raft_consensus.cc:2258] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:19.155581 3734 tablet_replica.cc:330] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: stopping tablet replica
I20260809 06:46:19.156454 3734 raft_consensus.cc:2229] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:19.157156 3734 raft_consensus.cc:2258] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:19.158802 3736 ts_tablet_manager.cc:1905] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:19.159374 3734 ts_tablet_manager.cc:1905] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:19.166304 3734 ts_tablet_manager.cc:1918] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:19.166546 3734 log.cc:1196] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/dcbcff934c87484e8fb2861e185df4c1
I20260809 06:46:19.167124 3734 ts_tablet_manager.cc:1939] T dcbcff934c87484e8fb2861e185df4c1 P 197d13cf275e492293b90dba861ad0ac: Deleting consensus metadata
I20260809 06:46:19.168946 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.145515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.27.209.193:45179)
0809 06:46:19.145713 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:19.145720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:19.164397 (+ 18677us) server_negotiation.cc:407] Connection header received
0809 06:46:19.164952 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:19.164960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:19.165097 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:19.165178 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:19.168048 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:19.168057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:19.168063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:19.168089 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:19.168135 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:19.168153 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:19.168259 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:19.168497 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 06:46:19.168570 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:46:19.169104 3024 catalog_manager.cc:4841] TS 197d13cf275e492293b90dba861ad0ac (127.27.209.195:44533): tablet dcbcff934c87484e8fb2861e185df4c1 (table test-table [id=88fe3b7c911c4dfa95ae3f2d210febb0]) successfully deleted
I20260809 06:46:19.170209 3306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:19.172967 3736 ts_tablet_manager.cc:1918] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:19.173177 3736 log.cc:1196] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/dcbcff934c87484e8fb2861e185df4c1
I20260809 06:46:19.173780 3736 ts_tablet_manager.cc:1939] T dcbcff934c87484e8fb2861e185df4c1 P af03f3867e294ebab0e2e3a3f274ca1c: Deleting consensus metadata
I20260809 06:46:19.176013 3025 catalog_manager.cc:4841] TS af03f3867e294ebab0e2e3a3f274ca1c (127.27.209.194:37953): tablet dcbcff934c87484e8fb2861e185df4c1 (table test-table [id=88fe3b7c911c4dfa95ae3f2d210febb0]) successfully deleted
I20260809 06:46:19.268162 3735 tablet_replica.cc:330] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113: stopping tablet replica
I20260809 06:46:19.268623 3735 raft_consensus.cc:2229] T dcbcff934c87484e8fb2861e185df4c1 P a2424aa81fbe49e7b870c4862f41f113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:19.268918 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 3129
W20260809 06:46:19.291952 3021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.209.193:45179 (error 108)
I20260809 06:46:19.292215 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 3315
I20260809 06:46:19.296533 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.293774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.27.209.194:37953)
0809 06:46:19.293967 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:19.293973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:19.293991 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:46:19.294073 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:19.294077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:19.294139 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:19.294240 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:19.295398 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:19.295405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:19.295411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:19.295432 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:19.295475 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:19.295492 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:19.295695 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:19.296122 (+ 427us) server_negotiation.cc:299] Negotiation successful
0809 06:46:19.296187 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:46:19.297452 3488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:19.299973 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.299095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36478)
0809 06:46:19.299580 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.299729 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
W20260809 06:46:19.300227 3021 catalog_manager.cc:4584] TS a2424aa81fbe49e7b870c4862f41f113 (127.27.209.193:45179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcbcff934c87484e8fb2861e185df4c1: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
I20260809 06:46:19.353922 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.353343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36494)
0809 06:46:19.353600 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.353735 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 06:46:19.355023 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.354599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36498)
0809 06:46:19.354759 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.354853 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 06:46:19.405946 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 3315
I20260809 06:46:19.415437 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.414994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36514)
0809 06:46:19.415184 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.415270 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 06:46:19.416538 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.416171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36528)
0809 06:46:19.416326 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.416399 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 06:46:19.427274 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 3499
I20260809 06:46:19.430008 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.428023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.27.209.195:44533)
0809 06:46:19.428203 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:19.428208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:19.428269 (+ 61us) server_negotiation.cc:407] Connection header received
0809 06:46:19.428423 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:19.428427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:19.428516 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:19.428597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:19.429474 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:19.429477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:19.429480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:19.429492 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:19.429517 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:19.429526 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:19.429629 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:19.429694 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 06:46:19.429736 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:46:19.430630 3663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:19.497225 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.496652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36544)
0809 06:46:19.496915 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.497044 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 06:46:19.498421 3744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:19.497943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45179 (local address 127.0.0.1:36550)
0809 06:46:19.498153 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:19.498243 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 06:46:19.514863 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 3499
I20260809 06:46:19.535020 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 2998
I20260809 06:46:19.536226 3086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:19.639803 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 2998
2026-08-09T06:46:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2835 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 06:46:19.682161 28487 mini_oidc.cc:152] Starting JWKS server
I20260809 06:46:19.684985 28487 webserver.cc:460] Webserver started at https://127.0.0.1:33183/ using document root <none> and password file <none>
I20260809 06:46:19.685249 28487 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 06:46:19.685896 28487 webserver.cc:460] Webserver started at http://127.0.0.1:43511/ using document root <none> and password file <none>
2026-08-09T06:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:19Z Disabled control of system clock
I20260809 06:46:19.714623 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36435
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:36435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/testchainca.cert with env {}
W20260809 06:46:20.033433 3753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:20.033736 3753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:20.033874 3753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:20.041435 3753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:20.041543 3753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:20.041604 3753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:20.041683 3753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:20.041745 3753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:46:20.041829 3753 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:46:20.041874 3753 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33183/jwks.json
I20260809 06:46:20.051733 3753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:36435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:20.053010 3753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:20.054337 3753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:20.065987 3766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:20.066045 3764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:20.066139 3763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:20.066733 3753 server_base.cc:1034] running on GCE node
I20260809 06:46:20.067514 3753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:20.068961 3753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:20.070209 3753 hybrid_clock.cc:648] HybridClock initialized: now 1786257980070148 us; error 51 us; skew 500 ppm
I20260809 06:46:20.070566 3753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:20.072850 3753 webserver.cc:460] Webserver started at http://127.27.209.254:45251/ using document root <none> and password file <none>
I20260809 06:46:20.073545 3753 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:20.073663 3753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:20.073964 3753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:20.076958 3753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "0147921bc60241359209b7aa4e1fd38a"
format_stamp: "Formatted at 2026-08-09 06:46:20 on dist-test-slave-jpj1"
server_key: "18751a85a84e80c6c6232ca520f8393b"
server_key_iv: "18ee5c5b4aa346aba8c302fe25810837"
server_key_version: "encryptionkey@0"
I20260809 06:46:20.077780 3753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "0147921bc60241359209b7aa4e1fd38a"
format_stamp: "Formatted at 2026-08-09 06:46:20 on dist-test-slave-jpj1"
server_key: "18751a85a84e80c6c6232ca520f8393b"
server_key_iv: "18ee5c5b4aa346aba8c302fe25810837"
server_key_version: "encryptionkey@0"
I20260809 06:46:20.083029 3753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:46:20.086673 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:20.088101 3753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:20.088305 3753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "0147921bc60241359209b7aa4e1fd38a"
format_stamp: "Formatted at 2026-08-09 06:46:20 on dist-test-slave-jpj1"
server_key: "18751a85a84e80c6c6232ca520f8393b"
server_key_iv: "18ee5c5b4aa346aba8c302fe25810837"
server_key_version: "encryptionkey@0"
I20260809 06:46:20.088512 3753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:20.114393 3753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:20.115219 3753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:20.115512 3753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:20.132586 3753 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:36435
I20260809 06:46:20.132606 3846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:36435 every 8 connection(s)
I20260809 06:46:20.133877 3753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:20.137737 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:20.140551 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 3753
I20260809 06:46:20.140745 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:20.141364 28487 external_mini_cluster.cc:1428] Setting key 325f30af8264aaecec09068f0ad21311
I20260809 06:46:20.150444 3849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a: Bootstrap starting.
I20260809 06:46:20.154210 3849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:20.155501 3849 log.cc:824] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:20.155921 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:20.143302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.27.209.254:36435)
0809 06:46:20.144221 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:20.144232 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:20.144273 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:46:20.145920 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:20.145949 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:20.146585 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:20.147038 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:20.147598 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.149530 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.150865 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.151892 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.152085 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:20.153102 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:20.153113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:20.153129 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:20.153821 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:20.153939 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:20.153966 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:20.154089 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:20.154277 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 06:46:20.154624 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":333,"threads_started":1}
I20260809 06:46:20.158653 3849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a: No bootstrap required, opened a new log
I20260809 06:46:20.159989 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:46:20.167181 3849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0147921bc60241359209b7aa4e1fd38a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36435 } }
I20260809 06:46:20.167513 3849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:20.167623 3849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0147921bc60241359209b7aa4e1fd38a, State: Initialized, Role: FOLLOWER
I20260809 06:46:20.168143 3849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0147921bc60241359209b7aa4e1fd38a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36435 } }
I20260809 06:46:20.168354 3849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:20.168452 3849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:20.168576 3849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:20.170622 3849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0147921bc60241359209b7aa4e1fd38a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36435 } }
I20260809 06:46:20.171072 3849 leader_election.cc:302] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0147921bc60241359209b7aa4e1fd38a; no voters:
I20260809 06:46:20.171610 3849 leader_election.cc:288] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:20.171759 3854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:20.172473 3854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [term 1 LEADER]: Becoming Leader. State: Replica: 0147921bc60241359209b7aa4e1fd38a, State: Running, Role: LEADER
I20260809 06:46:20.172962 3854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0147921bc60241359209b7aa4e1fd38a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36435 } }
I20260809 06:46:20.173651 3849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:20.175645 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0147921bc60241359209b7aa4e1fd38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0147921bc60241359209b7aa4e1fd38a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36435 } } }
I20260809 06:46:20.176123 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:20.175992 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0147921bc60241359209b7aa4e1fd38a. Latest consensus state: current_term: 1 leader_uuid: "0147921bc60241359209b7aa4e1fd38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0147921bc60241359209b7aa4e1fd38a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36435 } } }
I20260809 06:46:20.176249 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:20.177719 3868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:20.182621 3868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:20.192056 3868 catalog_manager.cc:1347] Generated new cluster ID: 16668370706f416b8dd04471fe885da3
I20260809 06:46:20.192320 3868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:20.211447 3868 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:20.212841 3868 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:20.225829 3868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0147921bc60241359209b7aa4e1fd38a: Generated new TSK 0
I20260809 06:46:20.226660 3868 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 06:46:20.266523 3895 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 06:46:20.266759 3895 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 06:46:20.269042 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:20.245444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.27.209.254:36435)
0809 06:46:20.245645 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:20.245652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:20.246104 (+ 452us) server_negotiation.cc:407] Connection header received
0809 06:46:20.246465 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:20.246471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:20.246562 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:20.246729 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:46:20.247255 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.248374 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.249842 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.250536 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.250678 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:20.251233 (+ 555us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:46:20.268322 (+ 17089us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 06:46:20.268704 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 06:46:20.269170 3852 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 06:46:20.270488 3893 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:20.245315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36435 (local address 127.0.0.1:43316)
0809 06:46:20.246006 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:20.246023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:20.246177 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:20.246785 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:20.246789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:20.246800 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:46:20.247111 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:20.247118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.248519 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.248524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:20.249671 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:20.249685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.250740 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.250745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:20.250874 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:20.250933 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:46:20.268621 (+ 17688us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 06:46:20.268920 (+ 299us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.209.254:36435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":341,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.209.254:36435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 06:46:20.272419 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 3753
I20260809 06:46:20.273573 3841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:20.366263 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 3753
2026-08-09T06:46:20Z chronyd exiting
I20260809 06:46:20.421173 28487 test_util.cc:175] -----------------------------------------------
I20260809 06:46:20.421334 28487 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786257935933888-28487-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (743 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 06:46:20.425228 28487 mini_oidc.cc:152] Starting JWKS server
I20260809 06:46:20.427070 28487 webserver.cc:460] Webserver started at https://127.0.0.1:45457/ using document root <none> and password file <none>
I20260809 06:46:20.427294 28487 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 06:46:20.428033 28487 webserver.cc:460] Webserver started at http://127.0.0.1:34965/ using document root <none> and password file <none>
2026-08-09T06:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:20Z Disabled control of system clock
I20260809 06:46:20.456475 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35349
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:33811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:35349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45457/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/testchainca.cert with env {}
W20260809 06:46:20.777345 3905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:20.777655 3905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:20.777789 3905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:20.784854 3905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:20.784971 3905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:20.785041 3905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:20.785126 3905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:20.785183 3905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:46:20.785267 3905 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:46:20.785319 3905 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45457/jwks.json
I20260809 06:46:20.795209 3905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:35349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45457/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:20.796562 3905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:20.798062 3905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:20.809892 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:20.809926 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:20.809962 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:20.810634 3905 server_base.cc:1034] running on GCE node
I20260809 06:46:20.811429 3905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:20.812855 3905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:20.814059 3905 hybrid_clock.cc:648] HybridClock initialized: now 1786257980814016 us; error 34 us; skew 500 ppm
I20260809 06:46:20.814414 3905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:20.820333 3905 webserver.cc:460] Webserver started at http://127.27.209.254:38855/ using document root <none> and password file <none>
I20260809 06:46:20.821048 3905 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:20.821163 3905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:20.821506 3905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:20.824569 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "7bca329f09144767ade26ca666d67891"
format_stamp: "Formatted at 2026-08-09 06:46:20 on dist-test-slave-jpj1"
server_key: "c73543b6ff11328f665d2e94ba537d7e"
server_key_iv: "a48298af5a1317c164895bc841e34ecb"
server_key_version: "encryptionkey@0"
I20260809 06:46:20.825347 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "7bca329f09144767ade26ca666d67891"
format_stamp: "Formatted at 2026-08-09 06:46:20 on dist-test-slave-jpj1"
server_key: "c73543b6ff11328f665d2e94ba537d7e"
server_key_iv: "a48298af5a1317c164895bc841e34ecb"
server_key_version: "encryptionkey@0"
I20260809 06:46:20.830458 3905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:46:20.833809 3921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:20.835070 3905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:46:20.835237 3905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "7bca329f09144767ade26ca666d67891"
format_stamp: "Formatted at 2026-08-09 06:46:20 on dist-test-slave-jpj1"
server_key: "c73543b6ff11328f665d2e94ba537d7e"
server_key_iv: "a48298af5a1317c164895bc841e34ecb"
server_key_version: "encryptionkey@0"
I20260809 06:46:20.835433 3905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:20.858700 3905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:20.859599 3905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:20.859912 3905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:20.877003 3905 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:35349
I20260809 06:46:20.877023 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:35349 every 8 connection(s)
I20260809 06:46:20.878368 3905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:20.882038 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:20.882701 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 3905
I20260809 06:46:20.882905 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:20.883760 28487 external_mini_cluster.cc:1428] Setting key ed1f699cd53b18a54c7704be90795754
I20260809 06:46:20.895519 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891: Bootstrap starting.
I20260809 06:46:20.896976 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:20.885693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.27.209.254:35349)
0809 06:46:20.886693 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:20.886712 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:20.886755 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:46:20.888366 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:20.888389 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:20.888853 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:20.889172 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:20.889739 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.890976 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.892754 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.893440 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.893586 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:20.894727 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:20.894735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:20.894748 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:20.895262 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:20.895341 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:20.895362 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:20.895445 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:20.895714 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 06:46:20.896040 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":368,"threads_started":1}
I20260809 06:46:20.898918 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:20.900105 4001 log.cc:824] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:20.900864 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:46:20.902961 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891: No bootstrap required, opened a new log
I20260809 06:46:20.910670 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bca329f09144767ade26ca666d67891" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35349 } }
I20260809 06:46:20.911020 4001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:20.911092 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bca329f09144767ade26ca666d67891, State: Initialized, Role: FOLLOWER
I20260809 06:46:20.911654 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bca329f09144767ade26ca666d67891" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35349 } }
I20260809 06:46:20.911948 4001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:20.912092 4001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:20.912272 4001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:20.914392 4001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bca329f09144767ade26ca666d67891" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35349 } }
I20260809 06:46:20.914865 4001 leader_election.cc:302] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bca329f09144767ade26ca666d67891; no voters:
I20260809 06:46:20.915401 4001 leader_election.cc:288] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:20.915558 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:20.916252 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [term 1 LEADER]: Becoming Leader. State: Replica: 7bca329f09144767ade26ca666d67891, State: Running, Role: LEADER
I20260809 06:46:20.916752 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bca329f09144767ade26ca666d67891" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35349 } }
I20260809 06:46:20.917508 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:20.919778 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bca329f09144767ade26ca666d67891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bca329f09144767ade26ca666d67891" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35349 } } }
I20260809 06:46:20.919785 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bca329f09144767ade26ca666d67891. Latest consensus state: current_term: 1 leader_uuid: "7bca329f09144767ade26ca666d67891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bca329f09144767ade26ca666d67891" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35349 } } }
I20260809 06:46:20.920207 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:20.920259 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:20.922415 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:20.927361 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:20.935691 4018 catalog_manager.cc:1347] Generated new cluster ID: 4d1f99ecf96a49c998ffa23d2a05447f
I20260809 06:46:20.935801 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:20.951996 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:20.953418 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:20.961308 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bca329f09144767ade26ca666d67891: Generated new TSK 0
I20260809 06:46:20.962112 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 06:46:20.993655 4041 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 06:46:20.993793 4041 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 06:46:20.994963 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:20.976291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.27.209.254:35349)
0809 06:46:20.976464 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:20.976471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:20.976859 (+ 388us) server_negotiation.cc:407] Connection header received
0809 06:46:20.977088 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:20.977096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:20.977168 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:20.977311 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:46:20.977794 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.978500 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.979877 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.980311 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.980432 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:20.980910 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:46:20.994268 (+ 13358us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 06:46:20.994637 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 06:46:20.995095 4006 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53026: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 06:46:20.995154 4040 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:20.976204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35349 (local address 127.0.0.1:53026)
0809 06:46:20.976754 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:20.976770 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:20.976909 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:20.977336 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:20.977341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:20.977353 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:46:20.977627 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:20.977636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.978636 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.978640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:20.979738 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:20.979751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:20.980473 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:20.980479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:20.980626 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:20.980659 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:46:20.994508 (+ 13849us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 06:46:20.994762 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.209.254:35349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45457/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":309,"threads_started":1}
I20260809 06:46:20.996536 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 3905
I20260809 06:46:20.997637 3995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:21.089864 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 3905
2026-08-09T06:46:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 06:46:21.148531 28487 mini_oidc.cc:152] Starting JWKS server
I20260809 06:46:21.150431 28487 webserver.cc:460] Webserver started at https://127.0.0.1:42845/ using document root <none> and password file <none>
I20260809 06:46:21.150672 28487 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 06:46:21.151274 28487 webserver.cc:460] Webserver started at http://127.0.0.1:33249/ using document root <none> and password file <none>
2026-08-09T06:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:21Z Disabled control of system clock
I20260809 06:46:21.180894 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:39067
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:45895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:39067
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 06:46:21.501924 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:21.502225 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:21.502342 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:21.509248 4050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:21.509356 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:21.509420 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:21.509509 4050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:21.509568 4050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:46:21.509672 4050 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:46:21.509727 4050 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42845/jwks.json
I20260809 06:46:21.518385 4050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:45895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:39067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:39067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:21.519618 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:21.521347 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:21.532543 4059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:21.532569 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:21.532615 4060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:21.533583 4050 server_base.cc:1034] running on GCE node
I20260809 06:46:21.534310 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:21.535607 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:21.536803 4050 hybrid_clock.cc:648] HybridClock initialized: now 1786257981536754 us; error 43 us; skew 500 ppm
I20260809 06:46:21.537124 4050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:21.539147 4050 webserver.cc:460] Webserver started at http://127.27.209.254:43837/ using document root <none> and password file <none>
I20260809 06:46:21.539848 4050 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:21.539950 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:21.540251 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:21.542966 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "ce5009e595964d65b2f0acf3139ef9d0"
format_stamp: "Formatted at 2026-08-09 06:46:21 on dist-test-slave-jpj1"
server_key: "0ccc095e4cf7fadf94595e17cd39bf7b"
server_key_iv: "fda06c47d491e40860b16ca87fd979f9"
server_key_version: "encryptionkey@0"
I20260809 06:46:21.543751 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "ce5009e595964d65b2f0acf3139ef9d0"
format_stamp: "Formatted at 2026-08-09 06:46:21 on dist-test-slave-jpj1"
server_key: "0ccc095e4cf7fadf94595e17cd39bf7b"
server_key_iv: "fda06c47d491e40860b16ca87fd979f9"
server_key_version: "encryptionkey@0"
I20260809 06:46:21.548547 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:46:21.551813 4069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:21.553006 4050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:21.553164 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "ce5009e595964d65b2f0acf3139ef9d0"
format_stamp: "Formatted at 2026-08-09 06:46:21 on dist-test-slave-jpj1"
server_key: "0ccc095e4cf7fadf94595e17cd39bf7b"
server_key_iv: "fda06c47d491e40860b16ca87fd979f9"
server_key_version: "encryptionkey@0"
I20260809 06:46:21.553361 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:21.569712 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:21.570451 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:21.570729 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:21.587343 4050 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:39067
I20260809 06:46:21.587416 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:39067 every 8 connection(s)
I20260809 06:46:21.588618 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:21.592247 4150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:21.597872 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 4050
I20260809 06:46:21.598104 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:21.598729 28487 external_mini_cluster.cc:1428] Setting key 26e6237466ddd0f5be73743de7139551
I20260809 06:46:21.603227 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0: Bootstrap starting.
I20260809 06:46:21.606618 4150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:21.607900 4150 log.cc:824] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:21.611136 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0: No bootstrap required, opened a new log
I20260809 06:46:21.612798 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:21.600786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.27.209.254:39067)
0809 06:46:21.601776 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:21.601803 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:21.601847 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:46:21.603511 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:21.603560 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:21.604018 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:21.604324 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:21.604931 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:21.606171 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:21.608002 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:21.608713 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:21.608864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:21.610012 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:21.610027 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:21.610045 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:21.610729 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:21.610827 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:21.610853 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:21.610962 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:21.611138 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 06:46:21.611511 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":363,"threads_started":1}
I20260809 06:46:21.616539 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:46:21.620091 4150 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5009e595964d65b2f0acf3139ef9d0" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39067 } }
I20260809 06:46:21.620433 4150 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:21.620536 4150 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5009e595964d65b2f0acf3139ef9d0, State: Initialized, Role: FOLLOWER
I20260809 06:46:21.621043 4150 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5009e595964d65b2f0acf3139ef9d0" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39067 } }
I20260809 06:46:21.621258 4150 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:21.621378 4150 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:21.621536 4150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:21.623591 4150 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5009e595964d65b2f0acf3139ef9d0" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39067 } }
I20260809 06:46:21.624132 4150 leader_election.cc:302] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5009e595964d65b2f0acf3139ef9d0; no voters:
I20260809 06:46:21.624734 4150 leader_election.cc:288] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:21.624866 4155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:21.626065 4155 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [term 1 LEADER]: Becoming Leader. State: Replica: ce5009e595964d65b2f0acf3139ef9d0, State: Running, Role: LEADER
I20260809 06:46:21.626554 4155 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5009e595964d65b2f0acf3139ef9d0" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39067 } }
I20260809 06:46:21.626935 4150 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:21.629109 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce5009e595964d65b2f0acf3139ef9d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5009e595964d65b2f0acf3139ef9d0" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39067 } } }
I20260809 06:46:21.629482 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:21.629920 4159 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce5009e595964d65b2f0acf3139ef9d0. Latest consensus state: current_term: 1 leader_uuid: "ce5009e595964d65b2f0acf3139ef9d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5009e595964d65b2f0acf3139ef9d0" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39067 } } }
I20260809 06:46:21.630247 4159 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:21.634200 4165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:21.639153 4165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:21.646534 4165 catalog_manager.cc:1347] Generated new cluster ID: 7dc5254ccc424efbac97ff9455dd0860
I20260809 06:46:21.646698 4165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:21.676288 4165 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:21.678053 4165 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:21.685525 4165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce5009e595964d65b2f0acf3139ef9d0: Generated new TSK 0
I20260809 06:46:21.686261 4165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:46:21.714465 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:21.706203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.27.209.254:39067)
0809 06:46:21.706391 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:21.706397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:21.706761 (+ 364us) server_negotiation.cc:407] Connection header received
0809 06:46:21.706975 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:21.706981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:21.707068 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:21.707248 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:46:21.707709 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:21.708836 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:21.710176 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:21.710846 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:21.710965 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:21.711421 (+ 456us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:46:21.713812 (+ 2391us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 06:46:21.714132 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41766: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 06:46:21.714596 4153 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41766: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 06:46:21.716076 4197 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:21.706090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:39067 (local address 127.0.0.1:41766)
0809 06:46:21.706671 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:21.706686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:21.706808 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:21.707250 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:21.707252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:21.707261 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:46:21.707568 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:21.707580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:21.708961 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:21.708964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:21.710045 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:21.710057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:21.711000 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:21.711004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:21.711123 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:21.711153 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:46:21.714011 (+ 2858us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 06:46:21.715661 (+ 1650us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.209.254:39067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42845/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1365,"thread_start_us":323,"threads_started":1}
I20260809 06:46:21.717478 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 4050
I20260809 06:46:21.718686 4145 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 06:46:21.725912 4198 webserver.cc:533] Webserver: sslize failed:
E20260809 06:46:21.725996 4198 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 06:46:21.808663 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 4050
2026-08-09T06:46:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T06:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:21Z Disabled control of system clock
I20260809 06:46:21.896763 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:34593
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:36445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:34593
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/testchainca.cert with env {}
W20260809 06:46:22.220543 4204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:22.220836 4204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:22.220964 4204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:22.228116 4204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:22.228219 4204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:22.228293 4204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:22.228358 4204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:22.228415 4204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 06:46:22.228498 4204 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 06:46:22.228546 4204 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 06:46:22.238299 4204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:34593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:22.239714 4204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:22.241024 4204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:22.252629 4213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:22.252669 4210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:22.252697 4211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:22.253386 4204 server_base.cc:1034] running on GCE node
I20260809 06:46:22.254237 4204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:22.255689 4204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:22.256898 4204 hybrid_clock.cc:648] HybridClock initialized: now 1786257982256841 us; error 51 us; skew 500 ppm
I20260809 06:46:22.257232 4204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:46:22.259455 4204 webserver.cc:460] Webserver started at http://127.27.209.254:40837/ using document root <none> and password file <none>
I20260809 06:46:22.260170 4204 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:22.260288 4204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:22.260591 4204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:22.263499 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "3b7738d2571343729abbfed7c42bd7aa"
format_stamp: "Formatted at 2026-08-09 06:46:22 on dist-test-slave-jpj1"
server_key: "292111c35be145df40fda65856591735"
server_key_iv: "e8fdc61a6b32c40c3f39d4ce33e0a8a2"
server_key_version: "encryptionkey@0"
I20260809 06:46:22.264365 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "3b7738d2571343729abbfed7c42bd7aa"
format_stamp: "Formatted at 2026-08-09 06:46:22 on dist-test-slave-jpj1"
server_key: "292111c35be145df40fda65856591735"
server_key_iv: "e8fdc61a6b32c40c3f39d4ce33e0a8a2"
server_key_version: "encryptionkey@0"
I20260809 06:46:22.269342 4204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:46:22.272743 4225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:22.274048 4204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:46:22.274225 4204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "3b7738d2571343729abbfed7c42bd7aa"
format_stamp: "Formatted at 2026-08-09 06:46:22 on dist-test-slave-jpj1"
server_key: "292111c35be145df40fda65856591735"
server_key_iv: "e8fdc61a6b32c40c3f39d4ce33e0a8a2"
server_key_version: "encryptionkey@0"
I20260809 06:46:22.274451 4204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:22.289762 4204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:22.290555 4204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:22.290817 4204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:22.307745 4204 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:34593
I20260809 06:46:22.307753 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:34593 every 8 connection(s)
I20260809 06:46:22.309027 4204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:22.312708 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:22.315454 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 4204
I20260809 06:46:22.315650 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:22.316277 28487 external_mini_cluster.cc:1428] Setting key 030b3be971cb6ff56ad78c727c733d1f
I20260809 06:46:22.326210 4297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa: Bootstrap starting.
I20260809 06:46:22.329939 4297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:22.331112 4297 log.cc:824] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:22.331446 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:22.318110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59602 (local address 127.27.209.254:34593)
0809 06:46:22.319132 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:22.319146 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:22.319196 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:46:22.321282 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:22.321311 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:22.322053 (+ 742us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:22.322522 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:22.323082 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:22.325055 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:22.326923 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:22.327640 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:22.327784 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:22.329013 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:22.329021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:22.329035 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:46:22.329567 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:22.329678 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:46:22.329707 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:22.329788 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:22.330071 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 06:46:22.330418 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":302,"threads_started":1}
I20260809 06:46:22.335290 4297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa: No bootstrap required, opened a new log
I20260809 06:46:22.335434 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:46:22.343124 4297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7738d2571343729abbfed7c42bd7aa" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34593 } }
I20260809 06:46:22.343506 4297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:22.343657 4297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7738d2571343729abbfed7c42bd7aa, State: Initialized, Role: FOLLOWER
I20260809 06:46:22.344317 4297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7738d2571343729abbfed7c42bd7aa" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34593 } }
I20260809 06:46:22.344558 4297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:22.344704 4297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:22.344897 4297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:22.347016 4297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7738d2571343729abbfed7c42bd7aa" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34593 } }
I20260809 06:46:22.347517 4297 leader_election.cc:302] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7738d2571343729abbfed7c42bd7aa; no voters:
I20260809 06:46:22.348107 4297 leader_election.cc:288] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:22.348223 4305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:22.348877 4305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [term 1 LEADER]: Becoming Leader. State: Replica: 3b7738d2571343729abbfed7c42bd7aa, State: Running, Role: LEADER
I20260809 06:46:22.349458 4305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7738d2571343729abbfed7c42bd7aa" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34593 } }
I20260809 06:46:22.350356 4297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:22.352429 4307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b7738d2571343729abbfed7c42bd7aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7738d2571343729abbfed7c42bd7aa" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34593 } } }
I20260809 06:46:22.352439 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b7738d2571343729abbfed7c42bd7aa. Latest consensus state: current_term: 1 leader_uuid: "3b7738d2571343729abbfed7c42bd7aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7738d2571343729abbfed7c42bd7aa" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34593 } } }
I20260809 06:46:22.352869 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:22.352869 4307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:22.356169 4313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:22.360844 4313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:22.369519 4313 catalog_manager.cc:1347] Generated new cluster ID: 729a480a51544704a39698b17f658d2b
I20260809 06:46:22.369659 4313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:22.389381 4313 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:22.391189 4313 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:22.397611 4313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b7738d2571343729abbfed7c42bd7aa: Generated new TSK 0
I20260809 06:46:22.398633 4313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:46:22.846829 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:22.420809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.27.209.254:34593)
0809 06:46:22.421031 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:22.421039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:22.421511 (+ 472us) server_negotiation.cc:407] Connection header received
0809 06:46:22.421726 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:22.421730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:22.421822 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:22.421944 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0809 06:46:22.422426 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:22.423143 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:22.425138 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:22.425660 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:22.425753 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:22.426334 (+ 581us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 06:46:22.846094 (+419760us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 06:46:22.846512 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59610: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 06:46:22.846985 4302 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59610: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 06:46:22.847107 4339 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:22.420783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:34593 (local address 127.0.0.1:59610)
0809 06:46:22.421404 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:22.421425 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:22.421579 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:22.421976 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:22.421980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:22.421994 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0809 06:46:22.422288 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:22.422299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:22.423290 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:22.423296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:22.424997 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:22.425012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:22.425862 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:22.425869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:22.426055 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:22.426114 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 06:46:22.846399 (+420285us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 06:46:22.846724 (+ 325us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.209.254:34593: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":310,"threads_started":1}
I20260809 06:46:22.848109 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 4204
I20260809 06:46:22.849287 4289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:22.951074 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 4204
2026-08-09T06:46:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1127 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1416 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (409 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:24 dist-test-slave-jpj1 krb5kdc[4366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:24 dist-test-slave-jpj1 krb5kdc[4366](info): set up 2 sockets
Aug 09 06:46:24 dist-test-slave-jpj1 krb5kdc[4366](info): commencing operation
krb5kdc: starting...
W20260809 06:46:25.965237 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:26 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:46:26.184861 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35739
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:35739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:26.508778 4387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:26.509040 4387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:26.509171 4387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:26.516613 4387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:26.516708 4387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:26.516785 4387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:26.516863 4387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:26.516927 4387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:26.526381 4387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:35739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:26.527662 4387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:26.529014 4387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:26.540573 4392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:26.540500 4395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:26.540529 4397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:26.541409 4387 server_base.cc:1034] running on GCE node
I20260809 06:46:26.542265 4387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:26.543659 4387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:26.544888 4387 hybrid_clock.cc:648] HybridClock initialized: now 1786257986544839 us; error 31 us; skew 500 ppm
I20260809 06:46:26.545204 4387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:26 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:26.550926 4387 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:26.552419 4387 webserver.cc:460] Webserver started at http://127.27.209.254:44589/ using document root <none> and password file <none>
I20260809 06:46:26.553071 4387 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:26.553174 4387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:26.553483 4387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:26.556367 4387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "3de1c5fcc01e4b758095b429a7455036"
format_stamp: "Formatted at 2026-08-09 06:46:26 on dist-test-slave-jpj1"
server_key: "5836f4df5cec6094361b5fdf2e86797d"
server_key_iv: "d5e5b47ece2c3838b2d9615b0516a2ac"
server_key_version: "encryptionkey@0"
I20260809 06:46:26.557081 4387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "3de1c5fcc01e4b758095b429a7455036"
format_stamp: "Formatted at 2026-08-09 06:46:26 on dist-test-slave-jpj1"
server_key: "5836f4df5cec6094361b5fdf2e86797d"
server_key_iv: "d5e5b47ece2c3838b2d9615b0516a2ac"
server_key_version: "encryptionkey@0"
I20260809 06:46:26.561962 4387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:46:26.565400 4407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:26.567011 4387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:46:26.567193 4387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "3de1c5fcc01e4b758095b429a7455036"
format_stamp: "Formatted at 2026-08-09 06:46:26 on dist-test-slave-jpj1"
server_key: "5836f4df5cec6094361b5fdf2e86797d"
server_key_iv: "d5e5b47ece2c3838b2d9615b0516a2ac"
server_key_version: "encryptionkey@0"
I20260809 06:46:26.567412 4387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:26.594435 4387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:26.599730 4387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:26.600106 4387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:26.620175 4387 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:35739
I20260809 06:46:26.620184 4478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:35739 every 8 connection(s)
I20260809 06:46:26.621893 4387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:26.625144 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 4387
I20260809 06:46:26.625396 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:26.625684 4483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:26.626084 28487 external_mini_cluster.cc:1428] Setting key 721cdef576c64abe1c3175f504ac5357
I20260809 06:46:26.639962 4483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036: Bootstrap starting.
Aug 09 06:46:26 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:26.643446 4483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:26.644539 4483 log.cc:824] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:26.647811 4483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036: No bootstrap required, opened a new log
I20260809 06:46:26.648561 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:26.627977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.27.209.254:35739)
0809 06:46:26.628827 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:26.628842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:26.628898 (+ 56us) server_negotiation.cc:407] Connection header received
0809 06:46:26.630580 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:26.630610 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:26.631143 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:26.631606 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:26.632926 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:26.634865 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:26.636168 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:26.637275 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:26.637473 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:26.641473 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:26.641481 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:26.641496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:26.641539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:26.644467 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:26.645323 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:26.645331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:26.645340 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:26.645558 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:26.645944 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:26.645948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:26.645950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:26.646448 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:26.646661 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:26.647160 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 06:46:26.647490 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":320,"threads_started":1}
I20260809 06:46:26.655230 4483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de1c5fcc01e4b758095b429a7455036" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35739 } }
I20260809 06:46:26.655664 4483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:26.655763 4483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de1c5fcc01e4b758095b429a7455036, State: Initialized, Role: FOLLOWER
I20260809 06:46:26.656347 4483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de1c5fcc01e4b758095b429a7455036" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35739 } }
I20260809 06:46:26.656579 4483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:26.656718 4483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:26.656893 4483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:26.659135 4483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de1c5fcc01e4b758095b429a7455036" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35739 } }
I20260809 06:46:26.659647 4483 leader_election.cc:302] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de1c5fcc01e4b758095b429a7455036; no voters:
I20260809 06:46:26.660221 4483 leader_election.cc:288] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:26.660368 4489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:26.661465 4489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [term 1 LEADER]: Becoming Leader. State: Replica: 3de1c5fcc01e4b758095b429a7455036, State: Running, Role: LEADER
I20260809 06:46:26.662035 4489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de1c5fcc01e4b758095b429a7455036" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35739 } }
I20260809 06:46:26.662319 4483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:26.664943 4490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de1c5fcc01e4b758095b429a7455036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de1c5fcc01e4b758095b429a7455036" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35739 } } }
I20260809 06:46:26.664980 4492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de1c5fcc01e4b758095b429a7455036. Latest consensus state: current_term: 1 leader_uuid: "3de1c5fcc01e4b758095b429a7455036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de1c5fcc01e4b758095b429a7455036" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35739 } } }
I20260809 06:46:26.665448 4490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:26.665494 4492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:26.667466 4502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:26.672673 4502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:26.681365 4502 catalog_manager.cc:1347] Generated new cluster ID: 29e6c8306b044a14af4f6009aa8ccdd5
I20260809 06:46:26.681474 4502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:26.706033 4502 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:26.707355 4502 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:26.721693 4502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de1c5fcc01e4b758095b429a7455036: Generated new TSK 0
I20260809 06:46:26.722537 4502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:46:26.776206 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35739
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:27.102985 4524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:27.103307 4524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:27.103446 4524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:27.110798 4524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:27.110929 4524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:27.111084 4524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:27.120945 4524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:27.122402 4524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:27.123854 4524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:27.136574 4531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:27.136595 4532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:27.136778 4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:27.137756 4524 server_base.cc:1034] running on GCE node
I20260809 06:46:27.138348 4524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:27.139252 4524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:27.140535 4524 hybrid_clock.cc:648] HybridClock initialized: now 1786257987140483 us; error 42 us; skew 500 ppm
I20260809 06:46:27.140894 4524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:27 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257987, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:27.144008 4524 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:46:27.145386 4524 webserver.cc:460] Webserver started at http://127.27.209.193:38063/ using document root <none> and password file <none>
I20260809 06:46:27.146044 4524 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:27.146162 4524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:27.146492 4524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:27.149808 4524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "ae813e65dc2642a0be1d634b23a76102"
format_stamp: "Formatted at 2026-08-09 06:46:27 on dist-test-slave-jpj1"
server_key: "9c1ef388cd2f1ff3f40a76c3adc94fe9"
server_key_iv: "869f25ce3eb5f9e67ba0a46e63be0752"
server_key_version: "encryptionkey@0"
I20260809 06:46:27.150569 4524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "ae813e65dc2642a0be1d634b23a76102"
format_stamp: "Formatted at 2026-08-09 06:46:27 on dist-test-slave-jpj1"
server_key: "9c1ef388cd2f1ff3f40a76c3adc94fe9"
server_key_iv: "869f25ce3eb5f9e67ba0a46e63be0752"
server_key_version: "encryptionkey@0"
I20260809 06:46:27.155609 4524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:46:27.158963 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:27.160351 4524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:27.160502 4524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "ae813e65dc2642a0be1d634b23a76102"
format_stamp: "Formatted at 2026-08-09 06:46:27 on dist-test-slave-jpj1"
server_key: "9c1ef388cd2f1ff3f40a76c3adc94fe9"
server_key_iv: "869f25ce3eb5f9e67ba0a46e63be0752"
server_key_version: "encryptionkey@0"
I20260809 06:46:27.160683 4524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:27.194896 4524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:27.198938 4524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:27.199210 4524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:27.200217 4524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:27.201541 4524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:27.201622 4524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:27.201759 4524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:27.201820 4524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:27.233995 4524 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:37329
I20260809 06:46:27.234069 4698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:37329 every 8 connection(s)
I20260809 06:46:27.235386 4524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:27.238374 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 4524
I20260809 06:46:27.238572 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:27.239418 28487 external_mini_cluster.cc:1428] Setting key b634d9a2e70535d9de205ce987e365c3
Aug 09 06:46:27 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257987, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:27.257280 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:27.237992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:50105 (local address 127.27.209.254:35739)
0809 06:46:27.238444 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:27.238450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:27.239742 (+ 1292us) server_negotiation.cc:407] Connection header received
0809 06:46:27.242019 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:27.242024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:27.242114 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:27.242209 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:27.243799 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.244520 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.246082 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.246707 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.246833 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:27.250988 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:27.250995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:27.251000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:27.251043 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:27.254193 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:27.254973 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:27.254979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:27.254983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:27.255149 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:27.255576 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:27.255580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:27.255583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:27.255908 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:27.256060 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:27.256742 (+ 682us) server_negotiation.cc:299] Negotiation successful
0809 06:46:27.256946 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":217,"threads_started":1}
I20260809 06:46:27.258145 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:27.238732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35739 (local address 127.27.209.193:50105)
0809 06:46:27.239568 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:27.239616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:27.241746 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:27.242472 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:27.242480 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:27.243019 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:27.243632 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:27.243654 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.244668 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.244674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:27.245939 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:27.245953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.246958 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.246965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:27.247249 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:27.247961 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:27.247995 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:27.250722 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:27.254341 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:27.254349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:27.254367 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:27.254830 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:27.255249 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:27.255252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:27.255254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:27.255442 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:27.256021 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:27.256027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:27.256352 (+ 325us) client_negotiation.cc:769] Sending connection context
0809 06:46:27.256588 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 06:46:27.257006 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":330,"threads_started":1}
I20260809 06:46:27.259707 4699 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35739
I20260809 06:46:27.260039 4699 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:27.260784 4699 heartbeater.cc:511] Master 127.27.209.254:35739 requested a full tablet report, sending...
I20260809 06:46:27.262912 4431 ts_manager.cc:194] Registered new tserver with Master: ae813e65dc2642a0be1d634b23a76102 (127.27.209.193:37329)
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
I20260809 06:46:27.264454 4431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:50105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:46:27.290539 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35739
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:27.618999 4710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:27.619321 4710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:27.619452 4710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:27.626642 4710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:27.626767 4710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:27.626922 4710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:27.636762 4710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:27.638054 4710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:27.639366 4710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:27.652128 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:27.652091 4723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:27.652670 4726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:27.653149 4710 server_base.cc:1034] running on GCE node
I20260809 06:46:27.653687 4710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:27.654573 4710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:27.655755 4710 hybrid_clock.cc:648] HybridClock initialized: now 1786257987655694 us; error 56 us; skew 500 ppm
I20260809 06:46:27.656023 4710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:27 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257987, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:27.658967 4710 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:46:27.660367 4710 webserver.cc:460] Webserver started at http://127.27.209.194:33777/ using document root <none> and password file <none>
I20260809 06:46:27.660979 4710 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:27.661078 4710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:27.661346 4710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:27.664317 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "b12178e341574a8e80392f4905605111"
format_stamp: "Formatted at 2026-08-09 06:46:27 on dist-test-slave-jpj1"
server_key: "f04335c0f9c23722ccd7c910c495ffa2"
server_key_iv: "448e4ef9cb5e7648ce95a581972f2336"
server_key_version: "encryptionkey@0"
I20260809 06:46:27.665071 4710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "b12178e341574a8e80392f4905605111"
format_stamp: "Formatted at 2026-08-09 06:46:27 on dist-test-slave-jpj1"
server_key: "f04335c0f9c23722ccd7c910c495ffa2"
server_key_iv: "448e4ef9cb5e7648ce95a581972f2336"
server_key_version: "encryptionkey@0"
I20260809 06:46:27.670164 4710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 06:46:27.673569 4738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:27.674822 4710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:46:27.674962 4710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "b12178e341574a8e80392f4905605111"
format_stamp: "Formatted at 2026-08-09 06:46:27 on dist-test-slave-jpj1"
server_key: "f04335c0f9c23722ccd7c910c495ffa2"
server_key_iv: "448e4ef9cb5e7648ce95a581972f2336"
server_key_version: "encryptionkey@0"
I20260809 06:46:27.675145 4710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:27.693584 4710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:27.697454 4710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:27.697718 4710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:27.698658 4710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:27.700016 4710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:27.700089 4710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:27.700223 4710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:27.700276 4710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:27.733009 4710 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:44545
I20260809 06:46:27.733058 4897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:44545 every 8 connection(s)
I20260809 06:46:27.734287 4710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:27.739727 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 4710
I20260809 06:46:27.739897 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:27.740422 28487 external_mini_cluster.cc:1428] Setting key da691fead3e81d08e6fde33aeebfd588
Aug 09 06:46:27 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257987, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:27.753718 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:27.736941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:51437 (local address 127.27.209.254:35739)
0809 06:46:27.737154 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:27.737160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:27.738322 (+ 1162us) server_negotiation.cc:407] Connection header received
0809 06:46:27.740154 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:27.740161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:27.740262 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:27.740376 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:27.741889 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.742589 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.744164 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.744594 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.744705 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:27.748230 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:27.748236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:27.748239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:27.748273 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:27.750610 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:27.751301 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:27.751305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:27.751307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:27.751433 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:27.751929 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:27.751933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:27.751935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:27.752209 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:27.752325 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:27.753225 (+ 900us) server_negotiation.cc:299] Negotiation successful
0809 06:46:27.753456 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:46:27.754962 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:27.737419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35739 (local address 127.27.209.194:51437)
0809 06:46:27.738152 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:27.738197 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:27.739858 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:27.740606 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:27.740614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:27.741150 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:27.741709 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:27.741728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.742738 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.742742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:27.744025 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:27.744038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:27.744823 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:27.744831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:27.745042 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:27.745663 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:27.745688 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:27.748014 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:27.750739 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:27.750748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:27.750765 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:27.751178 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:27.751564 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:27.751569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:27.751573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:27.751806 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:27.752337 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:27.752344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:27.752740 (+ 396us) client_negotiation.cc:769] Sending connection context
0809 06:46:27.753050 (+ 310us) client_negotiation.cc:240] Negotiation successful
0809 06:46:27.753586 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":288,"threads_started":1}
I20260809 06:46:27.756445 4898 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35739
I20260809 06:46:27.756768 4898 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:27.757584 4898 heartbeater.cc:511] Master 127.27.209.254:35739 requested a full tablet report, sending...
I20260809 06:46:27.759213 4431 ts_manager.cc:194] Registered new tserver with Master: b12178e341574a8e80392f4905605111 (127.27.209.194:44545)
I20260809 06:46:27.760072 4431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:51437
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:46:27.792598 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35739
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:28.111934 4905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:28.112221 4905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:28.112344 4905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:28.119457 4905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:28.119612 4905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:28.119772 4905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:28.129383 4905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:28.130784 4905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:28.132230 4905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:28.145325 4911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:28.145366 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:28.145401 4914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:28.146593 4905 server_base.cc:1034] running on GCE node
I20260809 06:46:28.147279 4905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:28.148324 4905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:28.149569 4905 hybrid_clock.cc:648] HybridClock initialized: now 1786257988149508 us; error 44 us; skew 500 ppm
I20260809 06:46:28.149956 4905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.153218 4905 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:46:28.154762 4905 webserver.cc:460] Webserver started at http://127.27.209.195:35063/ using document root <none> and password file <none>
I20260809 06:46:28.155485 4905 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:28.155642 4905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:28.155966 4905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:28.158952 4905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "66fbf6f89ea14f8c98dca4aae242b064"
format_stamp: "Formatted at 2026-08-09 06:46:28 on dist-test-slave-jpj1"
server_key: "735ee18e9cd4e0b97b65b2e544b08e08"
server_key_iv: "142ca0b3a269feb62119451fb13fb7f1"
server_key_version: "encryptionkey@0"
I20260809 06:46:28.159785 4905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "66fbf6f89ea14f8c98dca4aae242b064"
format_stamp: "Formatted at 2026-08-09 06:46:28 on dist-test-slave-jpj1"
server_key: "735ee18e9cd4e0b97b65b2e544b08e08"
server_key_iv: "142ca0b3a269feb62119451fb13fb7f1"
server_key_version: "encryptionkey@0"
I20260809 06:46:28.165184 4905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:46:28.169135 4929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:28.170647 4905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:46:28.170853 4905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "66fbf6f89ea14f8c98dca4aae242b064"
format_stamp: "Formatted at 2026-08-09 06:46:28 on dist-test-slave-jpj1"
server_key: "735ee18e9cd4e0b97b65b2e544b08e08"
server_key_iv: "142ca0b3a269feb62119451fb13fb7f1"
server_key_version: "encryptionkey@0"
I20260809 06:46:28.171068 4905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:28.184890 4905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:28.189167 4905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:28.189427 4905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:28.190398 4905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:28.192021 4905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:28.192117 4905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:28.192241 4905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:28.192303 4905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:28.222560 4905 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:41431
I20260809 06:46:28.222618 5085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:41431 every 8 connection(s)
I20260809 06:46:28.223853 4905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:28.232640 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 4905
I20260809 06:46:28.232798 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:28.233260 28487 external_mini_cluster.cc:1428] Setting key 5974cba4b6feca93514f98cf6e9aa422
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:28.243754 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.226323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:44705 (local address 127.27.209.254:35739)
0809 06:46:28.226491 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.226496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.227704 (+ 1208us) server_negotiation.cc:407] Connection header received
0809 06:46:28.229508 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.229514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.229607 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.229700 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.231187 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.231983 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.233778 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.234302 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.234426 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.238320 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.238324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.238327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.238361 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.240825 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.241509 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.241513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.241515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.241663 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.242077 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.242081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.242085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.242345 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.242475 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.243165 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.243385 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:46:28.244633 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.226736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35739 (local address 127.27.209.195:44705)
0809 06:46:28.227522 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:28.227589 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:28.229217 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:28.229944 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:28.229952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:28.230470 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:28.231015 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:28.231033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.232141 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.232145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:28.233592 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:28.233607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.234545 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.234553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:28.234848 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.235553 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:28.235588 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:28.238118 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:28.240965 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:28.240974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:28.240988 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:28.241394 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:28.241771 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:28.241776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:28.241780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:28.241967 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:28.242472 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:28.242480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:28.242796 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 06:46:28.243032 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 06:46:28.243567 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":292,"threads_started":1}
I20260809 06:46:28.245954 5086 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35739
I20260809 06:46:28.246260 5086 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:28.246970 5086 heartbeater.cc:511] Master 127.27.209.254:35739 requested a full tablet report, sending...
I20260809 06:46:28.248703 4431 ts_manager.cc:194] Registered new tserver with Master: 66fbf6f89ea14f8c98dca4aae242b064 (127.27.209.195:41431)
I20260809 06:46:28.249475 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:28.249650 4431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:44705
I20260809 06:46:28.267585 4699 heartbeater.cc:503] Master 127.27.209.254:35739 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/test-admin.keytab.
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.278947 28487 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 06:46:28.284404 5098 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:28.281745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.27.209.193:37329)
0809 06:46:28.282415 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.282422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.282446 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:46:28.282504 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.282511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.282727 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.282874 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.283976 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47662: BlockingRecv error: recv got EOF from 127.0.0.1:47662 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
W20260809 06:46:28.284453 28487 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc' of type FILE
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.285977 28487 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.287218 28487 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:28.301347 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.288772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.27.209.193:37329)
0809 06:46:28.288953 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.288959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.289066 (+ 107us) server_negotiation.cc:407] Connection header received
0809 06:46:28.289207 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.289211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.289270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.289349 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.290278 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.291332 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.292524 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.293100 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.293206 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.296374 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.296386 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.296401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.296432 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.299049 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.299645 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.299652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.299654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.299773 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.300058 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.300061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.300063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.300391 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.300600 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.300836 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.301070 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:46:28.306808 5098 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:28.305123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.27.209.193:37329)
0809 06:46:28.305301 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.305307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.305566 (+ 259us) server_negotiation.cc:407] Connection header received
0809 06:46:28.305723 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.305728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.305783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.305857 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.306569 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47676: BlockingRecv error: recv got EOF from 127.0.0.1:47676 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 06:46:28.307009 28487 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc' of type FILE
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.308492 28487 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.309742 28487 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:28.322911 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.310923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.27.209.193:37329)
0809 06:46:28.311095 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.311101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.311386 (+ 285us) server_negotiation.cc:407] Connection header received
0809 06:46:28.311523 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.311528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.311609 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.311691 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.312589 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.313273 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.314447 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.314894 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.315004 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.318291 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.318296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.318299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.318341 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.320615 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.321217 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.321221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.321223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.321343 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.321713 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.321717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.321719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.321972 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.322228 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.322471 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.322660 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 06:46:28.326890 5098 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:28.325301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.27.209.193:37329)
0809 06:46:28.325488 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.325494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.325722 (+ 228us) server_negotiation.cc:407] Connection header received
0809 06:46:28.325855 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.325861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.325931 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.326008 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.326676 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47698: BlockingRecv error: recv got EOF from 127.0.0.1:47698 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:46:28.327366 28487 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786257935933888-28487-0/krb5kdc/krb5cc)
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.328735 28487 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:28.329896 28487 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:28.342979 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.331034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.27.209.193:37329)
0809 06:46:28.331208 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.331214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.331499 (+ 285us) server_negotiation.cc:407] Connection header received
0809 06:46:28.331666 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.331669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.331719 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.331810 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.332673 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.333391 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.334522 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.334926 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.335028 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.338409 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.338414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.338417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.338446 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.340942 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.341475 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.341479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.341481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.341599 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.341867 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.341870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.341872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.342124 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.342339 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.342506 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.342666 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:46:28.344476 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 4524
I20260809 06:46:28.355436 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.345108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.27.209.193:37329)
0809 06:46:28.345253 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.345259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.345552 (+ 293us) server_negotiation.cc:407] Connection header received
0809 06:46:28.345684 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.345689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.345748 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.345823 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.346724 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.347359 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.348590 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.348984 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.349088 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.350950 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.350955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.350958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.350985 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.353096 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.353649 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.353654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.353656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.353797 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.354142 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.354145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.354147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.354391 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.354635 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.354925 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.355118 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 06:46:28.356056 4692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:28.437860 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 4524
I20260809 06:46:28.459141 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 4710
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:46:28.474011 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.460158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.27.209.194:44545)
0809 06:46:28.460713 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.460722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.460749 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:46:28.460814 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.460820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.461024 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.461156 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.462227 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.463332 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.464607 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.465159 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.465273 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.468856 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.468867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.468881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.468921 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.471515 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.472146 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.472154 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.472156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.472296 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.472577 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.472581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.472582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.472942 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.473172 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.473477 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.473712 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
I20260809 06:46:28.475369 4890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:28.556334 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 4710
I20260809 06:46:28.578531 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 4905
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[4366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:46:28.593312 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:28.579474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.27.209.195:41431)
0809 06:46:28.580105 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:28.580112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:28.580140 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:46:28.580201 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:28.580207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:28.580439 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:28.580564 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:28.581613 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.582607 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.583816 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:28.584367 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:28.584459 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:28.588129 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:28.588141 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:28.588156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:28.588187 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:28.590855 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.591415 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.591421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.591424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.591701 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:28.592004 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:28.592007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:28.592009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:28.592355 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:28.592580 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:28.592774 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 06:46:28.592993 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":331,"threads_started":1}
I20260809 06:46:28.594592 5080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:28.681593 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 4905
I20260809 06:46:28.703367 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 4387
I20260809 06:46:28.704594 4474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:28.806299 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 4387
2026-08-09T06:46:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4029 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[5129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[5129](info): set up 2 sockets
Aug 09 06:46:28 dist-test-slave-jpj1 krb5kdc[5129](info): commencing operation
krb5kdc: starting...
W20260809 06:46:30.002550 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:30 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab.
Entry for principal oryx/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab.
I20260809 06:46:30.233111 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35661
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab
--principal=oryx/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:30.555047 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:30.555343 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:30.555472 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:30.562722 5147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:30.562824 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:30.562896 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:30.562969 5147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:30.563026 5147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:30.572678 5147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:35661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab
--principal=oryx/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:30.573904 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:30.575194 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:30.586484 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:30.586584 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:30.586560 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:30.587464 5147 server_base.cc:1034] running on GCE node
I20260809 06:46:30.588266 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:30.589653 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:30.590843 5147 hybrid_clock.cc:648] HybridClock initialized: now 1786257990590788 us; error 50 us; skew 500 ppm
I20260809 06:46:30.591172 5147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:30 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:30.594240 5147 init.cc:377] Logged in from keytab as oryx/127.27.209.254@KRBTEST.COM (short username oryx)
I20260809 06:46:30.595760 5147 webserver.cc:460] Webserver started at http://127.27.209.254:39049/ using document root <none> and password file <none>
I20260809 06:46:30.596446 5147 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:30.596554 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:30.596846 5147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:30.599922 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "70dee95e74b741408369a89d11608f0b"
format_stamp: "Formatted at 2026-08-09 06:46:30 on dist-test-slave-jpj1"
server_key: "27b83e072711cf10e2727b58854a4613"
server_key_iv: "1f5466aa8b3367bfeafdbfb1d791f5a0"
server_key_version: "encryptionkey@0"
I20260809 06:46:30.600674 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "70dee95e74b741408369a89d11608f0b"
format_stamp: "Formatted at 2026-08-09 06:46:30 on dist-test-slave-jpj1"
server_key: "27b83e072711cf10e2727b58854a4613"
server_key_iv: "1f5466aa8b3367bfeafdbfb1d791f5a0"
server_key_version: "encryptionkey@0"
I20260809 06:46:30.605748 5147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:46:30.609117 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:30.610387 5147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:46:30.610545 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "70dee95e74b741408369a89d11608f0b"
format_stamp: "Formatted at 2026-08-09 06:46:30 on dist-test-slave-jpj1"
server_key: "27b83e072711cf10e2727b58854a4613"
server_key_iv: "1f5466aa8b3367bfeafdbfb1d791f5a0"
server_key_version: "encryptionkey@0"
I20260809 06:46:30.610720 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:30.643772 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:30.647923 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:30.648190 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:30.665395 5147 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:35661
I20260809 06:46:30.665428 5248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:35661 every 8 connection(s)
I20260809 06:46:30.666697 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:30.670388 5249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:30.674376 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 5147
I20260809 06:46:30.674589 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:30.675410 28487 external_mini_cluster.cc:1428] Setting key 0d92142d0d3be53ac8585172af606c39
I20260809 06:46:30.681980 5249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Bootstrap starting.
I20260809 06:46:30.685542 5249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:30.686666 5249 log.cc:824] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:30.689977 5249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: No bootstrap required, opened a new log
Aug 09 06:46:30 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:30.697597 5249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } }
I20260809 06:46:30.697927 5249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:30.698032 5249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70dee95e74b741408369a89d11608f0b, State: Initialized, Role: FOLLOWER
I20260809 06:46:30.698247 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:30.677504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.27.209.254:35661)
0809 06:46:30.678257 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:30.678267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:30.678308 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:46:30.680386 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:30.680409 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:30.680758 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:30.681074 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:30.682933 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:30.684312 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:30.686032 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:30.686673 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:30.686825 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:30.691280 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:30.691287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:30.691303 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:30.691340 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:30.694322 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:30.694950 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:30.694956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:30.694962 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:30.695129 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:30.695496 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:30.695499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:30.695502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:30.696093 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:30.696395 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:30.696836 (+ 441us) server_negotiation.cc:299] Negotiation successful
0809 06:46:30.697206 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":353,"threads_started":1}
I20260809 06:46:30.698654 5249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } }
I20260809 06:46:30.698909 5249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:30.699043 5249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:30.699216 5249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:30.701637 5249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } }
I20260809 06:46:30.702136 5249 leader_election.cc:302] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70dee95e74b741408369a89d11608f0b; no voters:
I20260809 06:46:30.702711 5249 leader_election.cc:288] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:30.702837 5256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:30.703498 5256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 LEADER]: Becoming Leader. State: Replica: 70dee95e74b741408369a89d11608f0b, State: Running, Role: LEADER
I20260809 06:46:30.704030 5256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } }
I20260809 06:46:30.704782 5249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:30.706828 5258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70dee95e74b741408369a89d11608f0b. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } }
I20260809 06:46:30.707917 5258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:30.706938 5257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } }
I20260809 06:46:30.708297 5257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:30.715678 5265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:30.719980 5265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:30.728571 5265 catalog_manager.cc:1347] Generated new cluster ID: dd41263487264d9f942755b4bead47d3
I20260809 06:46:30.728679 5265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:30.750260 5265 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:30.752203 5265 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:30.764549 5265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Generated new TSK 0
I20260809 06:46:30.765580 5265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab.
Entry for principal oryx/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab.
I20260809 06:46:30.827186 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35661
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:31.151216 5281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:31.151521 5281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:31.151682 5281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:31.158804 5281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:31.158929 5281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:31.159080 5281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:31.168633 5281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:31.169911 5281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:31.171196 5281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:31.184032 5293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:31.184226 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:31.184168 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:31.185343 5281 server_base.cc:1034] running on GCE node
I20260809 06:46:31.185905 5281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:31.186866 5281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:31.188062 5281 hybrid_clock.cc:648] HybridClock initialized: now 1786257991188003 us; error 49 us; skew 500 ppm
I20260809 06:46:31.188377 5281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:31 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257991, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:31.191432 5281 init.cc:377] Logged in from keytab as oryx/127.27.209.193@KRBTEST.COM (short username oryx)
I20260809 06:46:31.192819 5281 webserver.cc:460] Webserver started at http://127.27.209.193:46479/ using document root <none> and password file <none>
I20260809 06:46:31.193557 5281 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:31.193681 5281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:31.193985 5281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:31.197036 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "22ddf774e2214df7a6f8e0c3ebca0377"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "76bf7b5db2520f33cf9bd0fee83216af"
server_key_iv: "e9d9eb93597df3498921b98657016a9d"
server_key_version: "encryptionkey@0"
I20260809 06:46:31.197834 5281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "22ddf774e2214df7a6f8e0c3ebca0377"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "76bf7b5db2520f33cf9bd0fee83216af"
server_key_iv: "e9d9eb93597df3498921b98657016a9d"
server_key_version: "encryptionkey@0"
I20260809 06:46:31.202951 5281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:46:31.206501 5305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:31.208014 5281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:31.208180 5281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "22ddf774e2214df7a6f8e0c3ebca0377"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "76bf7b5db2520f33cf9bd0fee83216af"
server_key_iv: "e9d9eb93597df3498921b98657016a9d"
server_key_version: "encryptionkey@0"
I20260809 06:46:31.208364 5281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:31.237808 5281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:31.241900 5281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:31.242200 5281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:31.243209 5281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:31.244603 5281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:31.244692 5281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:31.244812 5281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:31.244868 5281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:31.274268 5281 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:34399
I20260809 06:46:31.274304 5471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:34399 every 8 connection(s)
I20260809 06:46:31.275488 5281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:31.280385 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 5281
I20260809 06:46:31.280591 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:31.281550 28487 external_mini_cluster.cc:1428] Setting key 5c95517798782519e5b1fad4c2183c85
Aug 09 06:46:31 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257991, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:31.297349 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:31.277998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:53557 (local address 127.27.209.254:35661)
0809 06:46:31.278427 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:31.278431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:31.279413 (+ 982us) server_negotiation.cc:407] Connection header received
0809 06:46:31.281945 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:31.281950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:31.282021 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:31.282112 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:31.283944 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.284651 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.286823 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.287213 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.287317 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:31.291320 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:31.291325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:31.291328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:31.291359 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:31.294309 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:31.295033 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:31.295038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:31.295042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:31.295185 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:31.295597 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:31.295602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:31.295604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:31.295950 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:31.296099 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:31.296775 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 06:46:31.296982 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260809 06:46:31.298086 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:31.278571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:53557)
0809 06:46:31.279261 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:31.279298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:31.281630 (+ 2332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:31.282505 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:31.282514 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:31.283064 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:31.283738 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:31.283767 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.284839 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.284845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:31.286665 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:31.286680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.287432 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.287438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:31.287737 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:31.288417 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:31.288446 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:31.291094 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:31.294448 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:31.294454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:31.294467 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:31.294909 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:31.295285 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:31.295288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:31.295290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:31.295469 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:31.296073 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:31.296079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:31.296381 (+ 302us) client_negotiation.cc:769] Sending connection context
0809 06:46:31.296602 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 06:46:31.296986 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":256,"threads_started":1}
I20260809 06:46:31.299590 5473 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35661
I20260809 06:46:31.299983 5473 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:31.300818 5473 heartbeater.cc:511] Master 127.27.209.254:35661 requested a full tablet report, sending...
I20260809 06:46:31.303128 5195 ts_manager.cc:194] Registered new tserver with Master: 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193:34399)
I20260809 06:46:31.304605 5195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:53557
WARNING: no policy specified for oryx/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab.
Entry for principal oryx/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab.
I20260809 06:46:31.334951 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35661
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:31.658556 5486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:31.658849 5486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:31.658977 5486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:31.666076 5486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:31.666186 5486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:31.666340 5486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:31.675592 5486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:31.676733 5486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:31.678022 5486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:31.690812 5491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:31.690785 5493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:31.690740 5495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:31.691525 5486 server_base.cc:1034] running on GCE node
I20260809 06:46:31.692143 5486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:31.693022 5486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:31.694226 5486 hybrid_clock.cc:648] HybridClock initialized: now 1786257991694163 us; error 43 us; skew 500 ppm
I20260809 06:46:31.694532 5486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:31 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257991, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:31.700275 5486 init.cc:377] Logged in from keytab as oryx/127.27.209.194@KRBTEST.COM (short username oryx)
I20260809 06:46:31.701648 5486 webserver.cc:460] Webserver started at http://127.27.209.194:44455/ using document root <none> and password file <none>
I20260809 06:46:31.702314 5486 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:31.702427 5486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:31.702718 5486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:31.705592 5486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "228c3285dfbd49eaa4a9978cdc0464db"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "0b9fae75409b10b1ae65ebea0f678fe9"
server_key_iv: "a4d872ef2c9dcf63dbe4967c34ee89c2"
server_key_version: "encryptionkey@0"
I20260809 06:46:31.706439 5486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "228c3285dfbd49eaa4a9978cdc0464db"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "0b9fae75409b10b1ae65ebea0f678fe9"
server_key_iv: "a4d872ef2c9dcf63dbe4967c34ee89c2"
server_key_version: "encryptionkey@0"
I20260809 06:46:31.711472 5486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:46:31.714993 5508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:31.716511 5486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:31.716657 5486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "228c3285dfbd49eaa4a9978cdc0464db"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "0b9fae75409b10b1ae65ebea0f678fe9"
server_key_iv: "a4d872ef2c9dcf63dbe4967c34ee89c2"
server_key_version: "encryptionkey@0"
I20260809 06:46:31.716845 5486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:31.742762 5486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:31.746932 5486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:31.747179 5486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:31.748183 5486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:31.749490 5486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:31.749563 5486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:31.749668 5486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:31.749719 5486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:31.779928 5486 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:45695
I20260809 06:46:31.779971 5670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:45695 every 8 connection(s)
I20260809 06:46:31.781123 5486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:31.786027 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 5486
I20260809 06:46:31.786201 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:31.786639 28487 external_mini_cluster.cc:1428] Setting key 21b5845f6ab13a9b844fc1c0254da5c3
Aug 09 06:46:31 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257991, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:31.802031 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:31.783828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:49427 (local address 127.27.209.254:35661)
0809 06:46:31.783994 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:31.783999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:31.785164 (+ 1165us) server_negotiation.cc:407] Connection header received
0809 06:46:31.786995 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:31.787001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:31.787083 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:31.787193 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:31.788945 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.789648 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.791838 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.792255 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.792363 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:31.796160 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:31.796166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:31.796170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:31.796209 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:31.798852 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:31.799722 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:31.799726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:31.799728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:31.799864 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:31.800344 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:31.800348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:31.800350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:31.800621 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:31.800759 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:31.801572 (+ 813us) server_negotiation.cc:299] Negotiation successful
0809 06:46:31.801774 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:46:31.803359 5675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:31.784279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:49427)
0809 06:46:31.784998 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:31.785042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:31.786705 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:31.787434 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:31.787445 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:31.788031 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:31.788762 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:31.788790 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.789811 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.789817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:31.791695 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:31.791711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:31.792482 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:31.792489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:31.792760 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:31.793451 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:31.793478 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:31.795917 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:31.799009 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:31.799018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:31.799034 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:31.799601 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:31.799963 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:31.799968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:31.799971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:31.800199 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:31.800753 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:31.800760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:31.801193 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 06:46:31.801486 (+ 293us) client_negotiation.cc:240] Negotiation successful
0809 06:46:31.801985 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":277,"threads_started":1}
I20260809 06:46:31.804982 5671 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35661
I20260809 06:46:31.805296 5671 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:31.806088 5671 heartbeater.cc:511] Master 127.27.209.254:35661 requested a full tablet report, sending...
I20260809 06:46:31.807600 5195 ts_manager.cc:194] Registered new tserver with Master: 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:31.808422 5195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:49427
WARNING: no policy specified for oryx/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab.
Entry for principal oryx/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab.
I20260809 06:46:31.838131 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35661
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:32.165738 5680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:32.166095 5680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:32.166253 5680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:32.173491 5680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:32.173604 5680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:32.173775 5680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:32.183653 5680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:32.184916 5680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:32.186228 5680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:32.199000 5686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:32.199043 5687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:32.199223 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:32.200030 5680 server_base.cc:1034] running on GCE node
I20260809 06:46:32.200681 5680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:32.201627 5680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:32.202829 5680 hybrid_clock.cc:648] HybridClock initialized: now 1786257992202772 us; error 53 us; skew 500 ppm
I20260809 06:46:32.203141 5680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:32.206367 5680 init.cc:377] Logged in from keytab as oryx/127.27.209.195@KRBTEST.COM (short username oryx)
I20260809 06:46:32.207897 5680 webserver.cc:460] Webserver started at http://127.27.209.195:32887/ using document root <none> and password file <none>
I20260809 06:46:32.208573 5680 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:32.208693 5680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:32.209004 5680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:32.212002 5680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "bbdce250244643f4968b48eab45d191c"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "c2abcfa4b650a3753222f3cd667b5234"
server_key_iv: "9f064f80e10ffe347ef22670f4113848"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.212841 5680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "bbdce250244643f4968b48eab45d191c"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "c2abcfa4b650a3753222f3cd667b5234"
server_key_iv: "9f064f80e10ffe347ef22670f4113848"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.218367 5680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:46:32.221879 5704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:32.223223 5680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 06:46:32.223381 5680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "bbdce250244643f4968b48eab45d191c"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "c2abcfa4b650a3753222f3cd667b5234"
server_key_iv: "9f064f80e10ffe347ef22670f4113848"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.223613 5680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:32.238405 5680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:32.242624 5680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:32.242986 5680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:32.244050 5680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:32.245496 5680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:32.245576 5680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:32.245697 5680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:32.245752 5680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:32.276971 5680 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:43853
I20260809 06:46:32.276989 5874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:43853 every 8 connection(s)
I20260809 06:46:32.278556 5680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:32.288388 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 5680
I20260809 06:46:32.288584 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:32.289072 28487 external_mini_cluster.cc:1428] Setting key e881e58e9c7a895f1808d9e74c51781e
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:32.297878 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.281337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:43251 (local address 127.27.209.254:35661)
0809 06:46:32.281554 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.281558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.282681 (+ 1123us) server_negotiation.cc:407] Connection header received
0809 06:46:32.284498 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.284503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.284583 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.284681 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.286224 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.286976 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.288782 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.289237 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.289349 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.292977 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.292982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.292985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.293016 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.295354 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.296038 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.296041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.296043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.296168 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.296534 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.296537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.296538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.296779 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.296882 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.297483 (+ 601us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.297640 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 06:46:32.298766 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.281749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.195:43251)
0809 06:46:32.282480 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.282519 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.284226 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.284929 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.284938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.285460 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.286042 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.286061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.287135 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.287139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.288636 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.288654 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.289471 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.289479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.289759 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.290386 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.290407 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.292799 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.295489 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.295496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.295508 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.295945 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.296259 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.296262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.296265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.296453 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.296870 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.296875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.297172 (+ 297us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.297379 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.297751 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":272,"threads_started":1}
I20260809 06:46:32.300051 5877 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35661
I20260809 06:46:32.300335 5877 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:32.301043 5877 heartbeater.cc:511] Master 127.27.209.254:35661 requested a full tablet report, sending...
I20260809 06:46:32.302460 5195 ts_manager.cc:194] Registered new tserver with Master: bbdce250244643f4968b48eab45d191c (127.27.209.195:43853)
I20260809 06:46:32.303373 5195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:43251
I20260809 06:46:32.304323 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:46:32.307844 5473 heartbeater.cc:503] Master 127.27.209.254:35661 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.27.209.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.253.keytab.
Entry for principal oryx/127.27.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.253.keytab.
I20260809 06:46:32.358513 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:33869
--webserver_interface=127.27.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.253.keytab
--principal=oryx/127.27.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:35661,127.27.209.253:33869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:32.693145 5882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:32.693426 5882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:32.693523 5882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:32.700661 5882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:32.700773 5882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:32.700826 5882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:32.700882 5882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:32.700927 5882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:32.710650 5882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:35661,127.27.209.253:33869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.253.keytab
--principal=oryx/127.27.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
W20260809 06:46:32.710948 5882 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.209.254:35661,127.27.209.253:33869'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 06:46:32.711948 5882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:32.713289 5882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:32.724521 5896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:32.724563 5893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:32.724583 5892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:32.725314 5882 server_base.cc:1034] running on GCE node
I20260809 06:46:32.726079 5882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:32.727464 5882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:32.728721 5882 hybrid_clock.cc:648] HybridClock initialized: now 1786257992728645 us; error 30 us; skew 500 ppm
I20260809 06:46:32.729049 5882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:32.732148 5882 init.cc:377] Logged in from keytab as oryx/127.27.209.253@KRBTEST.COM (short username oryx)
I20260809 06:46:32.733523 5882 webserver.cc:460] Webserver started at http://127.27.209.253:41691/ using document root <none> and password file <none>
I20260809 06:46:32.734213 5882 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:32.734319 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:32.734623 5882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:32.737706 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/instance:
uuid: "8094d282293f4843b5d01fb26cb0290b"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "dfe3891378a4b856d8ab9c27a4d3d03b"
server_key_iv: "53c55ff7f106256c715b04dfd2ae8c4d"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.738524 5882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal/instance:
uuid: "8094d282293f4843b5d01fb26cb0290b"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "dfe3891378a4b856d8ab9c27a4d3d03b"
server_key_iv: "53c55ff7f106256c715b04dfd2ae8c4d"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.743705 5882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:46:32.747056 5906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:32.748489 5882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:32.748651 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
uuid: "8094d282293f4843b5d01fb26cb0290b"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "dfe3891378a4b856d8ab9c27a4d3d03b"
server_key_iv: "53c55ff7f106256c715b04dfd2ae8c4d"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.748857 5882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:32.761909 5882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:32.765856 5882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:32.766088 5882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:32.783378 5882 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.253:33869
I20260809 06:46:32.783418 5984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.253:33869 every 8 connection(s)
I20260809 06:46:32.784771 5882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
I20260809 06:46:32.788273 5986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:32.789163 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 5882
I20260809 06:46:32.797509 5986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.253@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:32.810842 5671 heartbeater.cc:503] Master 127.27.209.254:35661 was elected leader, sending a full tablet report...
I20260809 06:46:32.813637 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.791430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.27.209.253:33869)
0809 06:46:32.792235 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.792249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.792315 (+ 66us) server_negotiation.cc:407] Connection header received
0809 06:46:32.794393 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.794422 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.794913 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.795371 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.796651 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.798607 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.799876 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.800728 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.801793 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.805674 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.805684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.805697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.805739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.808783 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.809542 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.809549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.809554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.809757 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.810116 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.810121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.810124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.810772 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.811061 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.811989 (+ 928us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.812393 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":332,"threads_started":1}
I20260809 06:46:32.815677 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.800394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:52684)
0809 06:46:32.801250 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.801302 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.801455 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.801861 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.801869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.802643 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.802978 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.802992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.804033 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.804037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.805379 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.805405 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.806403 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.806409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.806591 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.807137 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.807153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.809574 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.813169 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.813181 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.813185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.813604 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.814020 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.814024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.814028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.814230 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.814766 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.814773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.814947 (+ 174us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.815112 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.815324 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":481,"threads_started":1}
I20260809 06:46:32.816123 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.800356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.27.209.254:35661)
0809 06:46:32.800884 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.800890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.801400 (+ 510us) server_negotiation.cc:407] Connection header received
0809 06:46:32.801562 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.801568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.801650 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.801779 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.803119 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.803899 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.805528 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.806193 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.806338 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.809772 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.809779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.809783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.809821 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.813055 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.813723 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.813729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.813732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.813889 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.814345 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.814350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.814353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.814651 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.814776 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.815220 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.815471 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260809 06:46:32.817449 5986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.253@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
I20260809 06:46:32.828233 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.818155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44926)
0809 06:46:32.818301 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.818316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.818396 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.818609 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.818612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.818899 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.819024 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.819031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.819986 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.819989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.820906 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.820918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.821552 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.821556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.821680 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.822136 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.822151 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.823916 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.826314 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.826318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.826320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.826661 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.826952 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.826956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.826958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.827070 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.827615 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.827618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.827734 (+ 116us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.827844 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.828030 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 06:46:32.828474 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.818061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.27.209.253:33869)
0809 06:46:32.818216 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.818222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.818370 (+ 148us) server_negotiation.cc:407] Connection header received
0809 06:46:32.818468 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.818471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.818521 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.818589 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.819126 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.819860 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.821020 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.821430 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.821522 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.824070 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.824074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.824077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.824103 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.826192 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.826753 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.826757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.826759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.826862 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.827150 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.827153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.827155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.827493 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.827642 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.827923 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.828178 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":49}
I20260809 06:46:32.832868 5986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Bootstrap starting.
I20260809 06:46:32.835870 5986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:32.836838 5986 log.cc:824] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:32.839257 5986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: No bootstrap required, opened a new log
I20260809 06:46:32.846099 5986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } }
I20260809 06:46:32.846392 5986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:32.846477 5986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8094d282293f4843b5d01fb26cb0290b, State: Initialized, Role: FOLLOWER
I20260809 06:46:32.847030 5986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } }
I20260809 06:46:32.847858 5997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } } }
I20260809 06:46:32.848153 5997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:46:32.848731 5986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: configured and running, proceeding with master startup.
W20260809 06:46:32.856995 6012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:46:32.857111 6012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:46:32.859599 5882 master_runner.cc:428] Detected that this master 8094d282293f4843b5d01fb26cb0290b is joining an existing cluster
I20260809 06:46:32.859690 5882 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.209.253:33869
I20260809 06:46:32.879076 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.863244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.27.209.253:33869)
0809 06:46:32.863443 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.863450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.864759 (+ 1309us) server_negotiation.cc:407] Connection header received
0809 06:46:32.864890 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.864895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.864965 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.865089 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.866337 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.867193 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.869108 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.869796 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.869884 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.872316 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.872321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.872325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.872361 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.875842 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.876792 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.876797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.876799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.876938 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.877527 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.877532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.877535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.877901 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.878058 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.878511 (+ 453us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.878735 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:46:32.880340 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.863873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44938)
0809 06:46:32.864659 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.864676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.864788 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.865117 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.865122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.865848 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.866165 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.866174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.867392 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.867396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.868945 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.868962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.870166 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.870171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.870330 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.870981 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.871003 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.872137 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.875971 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.875975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.875978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.876679 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.877047 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.877050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.877054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.877206 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.878121 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.878125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.878275 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.879599 (+ 1324us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.879868 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":68,"thread_start_us":384,"threads_started":1}
I20260809 06:46:32.882015 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.863283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.27.209.254:35661)
0809 06:46:32.863477 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.863483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.864166 (+ 683us) server_negotiation.cc:407] Connection header received
0809 06:46:32.864392 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.864398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.864494 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.864570 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.866119 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.867246 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.869387 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.870036 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.870133 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.875631 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.875638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.875642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.875683 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.878971 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.879741 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.879746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.879750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.879903 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.880395 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.880400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.880404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.880747 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.880874 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.881456 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.881678 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:46:32.882088 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.863305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:52694)
0809 06:46:32.864064 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.864080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.864219 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.864800 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.864804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.865520 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.865936 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.865952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.867419 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.867425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.869203 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.869222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.870445 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.870453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.870980 (+ 527us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.871615 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.871638 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.875376 (+ 3738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.879118 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.879123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.879127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.879601 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.880048 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.880052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.880056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.880247 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.880958 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.880962 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.881158 (+ 196us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.881323 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.881654 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":466,"threads_started":1}
WARNING: no policy specified for oryx/127.27.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.252@KRBTEST.COM" created.
Aug 09 06:46:32 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.254@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
I20260809 06:46:32.906232 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.888892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44952)
0809 06:46:32.889534 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.889574 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.889878 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.890596 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.890604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.891521 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.891901 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.891914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.893347 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.893353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.894948 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.894964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.896033 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.896038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.896188 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.896796 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.896820 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.899379 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.903141 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.903155 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.903163 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.903812 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.904247 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.904252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.904255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.904410 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.905109 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.905118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.905317 (+ 199us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.905511 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.905848 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":483,"threads_started":1}
I20260809 06:46:32.907173 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.889182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.27.209.253:33869)
0809 06:46:32.889367 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.889373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.889684 (+ 311us) server_negotiation.cc:407] Connection header received
0809 06:46:32.890127 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.890132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.890233 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.890304 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.892075 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.893198 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.895092 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.895843 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.895980 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.899613 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.899619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.899627 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.899667 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.902997 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.903937 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.903942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.903945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.904104 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.904538 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.904543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.904546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.904972 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.905122 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.906735 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.906915 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:46:32.908951 5195 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.209.253:33869: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8094d282293f4843b5d01fb26cb0290b"
member_type: NON_VOTER
last_known_addr {
host: "127.27.209.253"
port: 33869
}
attrs {
promote: true
}
}
dest_uuid: "70dee95e74b741408369a89d11608f0b"
cas_config_opid_index: -1
I20260809 06:46:32.910001 5195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } }
I20260809 06:46:32.925135 6031 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8094d282293f4843b5d01fb26cb0290b (127.27.209.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } }
I20260809 06:46:32.926774 5948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:32.929790 6031 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.209.253:33869
I20260809 06:46:32.930573 5948 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Refusing update from remote peer 70dee95e74b741408369a89d11608f0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 06:46:32.930951 5882 master.cc:537] Master@127.27.209.253:33869 shutting down...
I20260809 06:46:32.931630 6030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:32.931002 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } }
I20260809 06:46:32.932365 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.252.keytab.
Entry for principal oryx/127.27.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.252.keytab.
I20260809 06:46:32.935763 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:36221
--webserver_interface=127.27.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.252.keytab
--principal=oryx/127.27.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:32.936012 5167 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260809 06:46:32.946198 5882 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:32.946842 5882 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:32.947173 5882 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: stopping tablet replica
I20260809 06:46:32.955477 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.955013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44956)
0809 06:46:32.955210 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.955321 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:33869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 06:46:32.955811 5167 consensus_peers.cc:487] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b -> Peer 8094d282293f4843b5d01fb26cb0290b (127.27.209.253:33869): Couldn't send request to peer 8094d282293f4843b5d01fb26cb0290b. Status: Network error: Client connection negotiation failed: client connection to 127.27.209.253:33869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 06:46:32.965838 5882 master.cc:552] Master@127.27.209.253:33869 shutdown complete.
I20260809 06:46:32.965969 5882 master_runner.cc:305] Clearing existing system tablet
I20260809 06:46:32.966378 5882 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:32.966480 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:32.968838 5882 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 06:46:32.970202 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:32.970487 5882 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:32.970738 5882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 06:46:32.970832 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
uuid: "8094d282293f4843b5d01fb26cb0290b"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "dfe3891378a4b856d8ab9c27a4d3d03b"
server_key_iv: "53c55ff7f106256c715b04dfd2ae8c4d"
server_key_version: "encryptionkey@0"
I20260809 06:46:32.974186 5882 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:32.978941 5882 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 06:46:32.979107 5882 log.cc:1196] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 06:46:32.979486 5882 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Deleting consensus metadata
I20260809 06:46:32.979808 5882 master_runner.cc:315] Copying system tablet from 127.27.209.254:35661
I20260809 06:46:32.982211 5882 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: tablet copy: Beginning tablet copy session from remote peer at address 127.27.209.254:35661
I20260809 06:46:32.994611 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.982933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.27.209.254:35661)
0809 06:46:32.983146 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:32.983153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:32.983614 (+ 461us) server_negotiation.cc:407] Connection header received
0809 06:46:32.983825 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:32.983831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:32.983901 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:32.983985 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:32.985120 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.985853 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.987084 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.987521 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.987676 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.989425 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:32.989430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:32.989433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:32.989465 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:32.992065 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.992672 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.992676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.992678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.992800 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:32.993184 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:32.993189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:32.993192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:32.993453 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:32.993647 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:32.994011 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 06:46:32.994219 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:46:32.994637 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:32.982898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:52698)
0809 06:46:32.983483 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:32.983501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:32.983668 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:32.984041 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:32.984046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:32.984492 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:32.984710 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.984720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.986009 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.986015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.986928 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:32.986940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:32.987701 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:32.987707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:32.987883 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:32.988483 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:32.988500 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:32.989229 (+ 729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:32.992195 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.992200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.992205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.992559 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.992904 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:32.992909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:32.992913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:32.993064 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:32.993571 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:32.993576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:32.993754 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 06:46:32.993890 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 06:46:32.994206 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":371,"threads_started":1}
I20260809 06:46:32.995934 5224 tablet_copy_service.cc:140] P 70dee95e74b741408369a89d11608f0b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8094d282293f4843b5d01fb26cb0290b ({username='oryx', principal='oryx/127.27.209.253@KRBTEST.COM'} at 127.0.0.1:52698)
I20260809 06:46:32.996169 5224 tablet_copy_service.cc:161] P 70dee95e74b741408369a89d11608f0b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8094d282293f4843b5d01fb26cb0290b at {username='oryx', principal='oryx/127.27.209.253@KRBTEST.COM'} at 127.0.0.1:52698: session id = 8094d282293f4843b5d01fb26cb0290b-00000000000000000000000000000000
I20260809 06:46:32.999281 5224 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 06:46:33.001161 5882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 06:46:33.006156 5882 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: tablet copy: Starting download of 0 data blocks...
I20260809 06:46:33.006438 5882 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: tablet copy: Starting download of 1 WAL segments...
I20260809 06:46:33.008838 5882 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 06:46:33.011932 5224 tablet_copy_service.cc:342] P 70dee95e74b741408369a89d11608f0b: Request end of tablet copy session 8094d282293f4843b5d01fb26cb0290b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.209.253@KRBTEST.COM'} at 127.0.0.1:52698
I20260809 06:46:33.012149 5224 tablet_copy_service.cc:433] P 70dee95e74b741408369a89d11608f0b: ending tablet copy session 8094d282293f4843b5d01fb26cb0290b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8094d282293f4843b5d01fb26cb0290b
I20260809 06:46:33.022384 5882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:33.024986 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:33.034933 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:33.035919 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:33.040833 5882 server_base.cc:1034] running on GCE node
I20260809 06:46:33.041175 5882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:33.041837 5882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:33.043139 5882 hybrid_clock.cc:648] HybridClock initialized: now 1786257993043070 us; error 35 us; skew 500 ppm
I20260809 06:46:33.043380 5882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:33.045305 5882 init.cc:377] Logged in from keytab as oryx/127.27.209.253@KRBTEST.COM (short username oryx)
I20260809 06:46:33.046653 5882 webserver.cc:460] Webserver started at http://127.27.209.253:35263/ using document root <none> and password file <none>
I20260809 06:46:33.046980 5882 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:33.047053 5882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:33.049491 5882 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 06:46:33.051139 6060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:33.051575 5882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260809 06:46:33.051728 5882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
uuid: "8094d282293f4843b5d01fb26cb0290b"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "dfe3891378a4b856d8ab9c27a4d3d03b"
server_key_iv: "53c55ff7f106256c715b04dfd2ae8c4d"
server_key_version: "encryptionkey@0"
I20260809 06:46:33.051899 5882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:33.066833 5882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:33.070150 5882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:33.094058 5882 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.253:33869
I20260809 06:46:33.094084 6126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.253:33869 every 8 connection(s)
I20260809 06:46:33.095016 5882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
I20260809 06:46:33.100684 6127 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:46:33.103757 6127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Bootstrap starting.
I20260809 06:46:33.116868 6127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:46:33.117305 6127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: Bootstrap complete.
I20260809 06:46:33.118120 6127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } }
I20260809 06:46:33.118340 6127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8094d282293f4843b5d01fb26cb0290b, State: Initialized, Role: LEARNER
I20260809 06:46:33.118567 6127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } }
I20260809 06:46:33.119295 6130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } }
I20260809 06:46:33.119625 6127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:33.119712 6130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: LEARNER
I20260809 06:46:33.133874 6143 catalog_manager.cc:1259] Loaded cluster ID: dd41263487264d9f942755b4bead47d3
I20260809 06:46:33.133984 6143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: loading cluster ID for follower catalog manager: success
I20260809 06:46:33.136482 6143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: acquiring CA information for follower catalog manager: success
I20260809 06:46:33.138433 6143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:46:33.306037 5877 heartbeater.cc:503] Master 127.27.209.254:35661 was elected leader, sending a full tablet report...
W20260809 06:46:33.351603 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:33.351955 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:33.352100 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:33.359645 6035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:33.359773 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:33.359851 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:33.359925 6035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:33.359989 6035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:33.369916 6035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.252.keytab
--principal=oryx/127.27.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:36221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:33.371251 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:33.372649 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:33.384256 6157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:33.384320 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:33.384720 6153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:33.385070 6035 server_base.cc:1034] running on GCE node
I20260809 06:46:33.385865 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:33.387203 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:33.388437 6035 hybrid_clock.cc:648] HybridClock initialized: now 1786257993388378 us; error 41 us; skew 500 ppm
I20260809 06:46:33.388796 6035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:33.391942 6035 init.cc:377] Logged in from keytab as oryx/127.27.209.252@KRBTEST.COM (short username oryx)
I20260809 06:46:33.393393 6035 webserver.cc:460] Webserver started at http://127.27.209.252:45187/ using document root <none> and password file <none>
I20260809 06:46:33.394065 6035 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:33.394192 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:33.394495 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:33.397487 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/instance:
uuid: "85f7189c52a64f1bb9c06f2bce3196f5"
format_stamp: "Formatted at 2026-08-09 06:46:33 on dist-test-slave-jpj1"
server_key: "653ac8b9357a9217e46cf981def772b4"
server_key_iv: "7278620552df2637f03e29949ff36099"
server_key_version: "encryptionkey@0"
I20260809 06:46:33.398233 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal/instance:
uuid: "85f7189c52a64f1bb9c06f2bce3196f5"
format_stamp: "Formatted at 2026-08-09 06:46:33 on dist-test-slave-jpj1"
server_key: "653ac8b9357a9217e46cf981def772b4"
server_key_iv: "7278620552df2637f03e29949ff36099"
server_key_version: "encryptionkey@0"
I20260809 06:46:33.403980 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:46:33.408414 6168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:33.409767 6035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:46:33.409930 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
uuid: "85f7189c52a64f1bb9c06f2bce3196f5"
format_stamp: "Formatted at 2026-08-09 06:46:33 on dist-test-slave-jpj1"
server_key: "653ac8b9357a9217e46cf981def772b4"
server_key_iv: "7278620552df2637f03e29949ff36099"
server_key_version: "encryptionkey@0"
I20260809 06:46:33.410107 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:33.423933 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.417895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44964)
0809 06:46:33.418068 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.418085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.418202 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.418912 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.418918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.418949 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:33.419125 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.419134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.420610 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.420616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.421927 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.421941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.423090 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.423096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.423318 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.423379 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.423466 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.423634 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 06:46:33.423961 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.418059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.27.209.253:33869)
0809 06:46:33.418586 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.418590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.418609 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:46:33.418669 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.418673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.418741 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.418849 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:33.419265 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.420463 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.422084 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.422920 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.423097 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.423301 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.423503 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.423659 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":283,"threads_started":1}
I20260809 06:46:33.425320 6096 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 06:46:33.425864 6130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70dee95e74b741408369a89d11608f0b. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } }
I20260809 06:46:33.426149 6130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: LEARNER
I20260809 06:46:33.437342 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:33.441039 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:33.441278 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:33.458655 6035 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.252:36221
I20260809 06:46:33.458698 6244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.252:36221 every 8 connection(s)
I20260809 06:46:33.460013 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
I20260809 06:46:33.463950 6245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:33.466274 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 6035
I20260809 06:46:33.471454 6245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.252@KRBTEST.COM
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.252@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:33.486865 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.473929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.27.209.254:35661)
0809 06:46:33.474111 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.474116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.474593 (+ 477us) server_negotiation.cc:407] Connection header received
0809 06:46:33.474782 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.474786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.474850 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.474931 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.476072 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.476738 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.478135 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.478513 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.478619 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.481871 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.481875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.481878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.481909 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.484417 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.485083 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.485086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.485089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.485208 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.485554 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.485559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.485562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.485831 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.485973 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.486425 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.486594 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:46:33.487416 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.473752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:52714)
0809 06:46:33.474451 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.474499 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.474651 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.475005 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.475012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.475662 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.475928 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.475938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.476880 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.476884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.477996 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.478009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.478732 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.478740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.478877 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.479421 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.479447 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.481689 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.484539 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.484550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.484557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.484972 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.485313 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.485318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.485321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.485455 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.485964 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.485972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.486150 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.486360 (+ 210us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.486656 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":317,"threads_started":1}
I20260809 06:46:33.487435 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.468440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.27.209.252:36221)
0809 06:46:33.469197 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.469208 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.469251 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:46:33.470864 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.470885 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.471253 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.471618 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.473068 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.474563 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.475833 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.476444 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.476601 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.480653 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.480661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.480671 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.480703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.483261 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.483931 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.483937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.483942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.484080 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.484442 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.484446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.484448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.484962 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.485243 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.485657 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.486088 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"spinlock_wait_cycles":12544,"thread_start_us":323,"threads_started":1}
I20260809 06:46:33.489799 6245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.252@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
I20260809 06:46:33.501456 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.490557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44970)
0809 06:46:33.490723 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.490743 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.490834 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.491107 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.491117 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.491442 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.491603 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.491612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.492552 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.492557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.493480 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.493492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.494191 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.494197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.494351 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.494820 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.494836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.496718 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.499227 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.499232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.499236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.499653 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.499991 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.499996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.500000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.500136 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.500619 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.500624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.500757 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.500863 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.501127 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 06:46:33.501513 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.490671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.27.209.253:33869)
0809 06:46:33.490846 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.490852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.490870 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:46:33.490928 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.490933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.490999 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.491076 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.491735 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.492431 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.493630 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.494039 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.494147 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.496896 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.496901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.496904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.496934 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.499101 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.499757 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.499762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.499765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.499893 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.500249 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.500254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.500258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.500508 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.500636 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.500984 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.501190 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:46:33.502609 6245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.252@KRBTEST.COM for oryx/127.27.209.252@KRBTEST.COM
I20260809 06:46:33.514107 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.503354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36556)
0809 06:46:33.503513 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.503530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.503655 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.503958 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.503962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.504294 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.504417 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.504426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.505393 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.505398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.506296 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.506308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.506937 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.506943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.507123 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.507571 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.507588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.509385 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.511863 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.511868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.511872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.512223 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.512546 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.512551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.512555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.512703 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.513155 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.513160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.513317 (+ 157us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.513454 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.513693 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 06:46:33.514178 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.503336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.27.209.252:36221)
0809 06:46:33.503513 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.503518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.503611 (+ 93us) server_negotiation.cc:407] Connection header received
0809 06:46:33.503770 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.503775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.503855 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.503930 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.504540 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.505278 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.506417 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.506811 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.506916 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.509547 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.509551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.509554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.509592 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.511745 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.512323 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.512328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.512331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.512439 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.512800 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.512804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.512807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.513055 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.513183 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.513541 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.513835 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":65}
I20260809 06:46:33.518636 6245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Bootstrap starting.
I20260809 06:46:33.521936 6245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:33.522899 6245 log.cc:824] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:33.525378 6245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: No bootstrap required, opened a new log
I20260809 06:46:33.532577 6245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } }
I20260809 06:46:33.532883 6245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:33.532970 6245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f7189c52a64f1bb9c06f2bce3196f5, State: Initialized, Role: FOLLOWER
I20260809 06:46:33.533468 6245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } }
I20260809 06:46:33.534256 6254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } } }
I20260809 06:46:33.534548 6254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:46:33.535180 6245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:33.542315 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 5281
W20260809 06:46:33.543886 6271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:46:33.544020 6271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:46:33.549057 6035 master_runner.cc:428] Detected that this master 85f7189c52a64f1bb9c06f2bce3196f5 is joining an existing cluster
I20260809 06:46:33.549177 6035 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.209.252:36221
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.193@KRBTEST.COM
I20260809 06:46:33.563870 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.544109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.27.209.193:34399)
0809 06:46:33.544717 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.544723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.544746 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:46:33.544805 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.544813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.545059 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.545220 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.546861 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.548589 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.550043 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.550847 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.550966 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.555913 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.555927 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.555942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.555987 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.559749 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.560614 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.560622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.560626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.560781 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.561281 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.561286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.561290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.561712 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.563013 (+ 1301us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.563174 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.563440 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":303,"threads_started":1}
I20260809 06:46:33.565557 5465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:33.570442 6254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:33.570783 6254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } }
I20260809 06:46:33.572503 6254 leader_election.cc:288] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661), 8094d282293f4843b5d01fb26cb0290b (127.27.209.253:33869)
I20260809 06:46:33.574026 5206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70dee95e74b741408369a89d11608f0b" is_pre_election: true
I20260809 06:46:33.574875 6096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8094d282293f4843b5d01fb26cb0290b" is_pre_election: true
I20260809 06:46:33.576234 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.555327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.27.209.254:35661)
0809 06:46:33.555503 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.555509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.555851 (+ 342us) server_negotiation.cc:407] Connection header received
0809 06:46:33.556137 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.556143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.556225 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.556297 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.557800 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.558888 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.560941 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.561534 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.561629 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.564540 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.564546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.564550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.564586 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.568077 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.572783 (+ 4706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.572788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.572792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.572954 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.573822 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.573827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.573830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.574166 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.574405 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.575661 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.575891 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:46:33.576224 6173 leader_election.cc:302] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85f7189c52a64f1bb9c06f2bce3196f5; no voters: 70dee95e74b741408369a89d11608f0b, 8094d282293f4843b5d01fb26cb0290b
I20260809 06:46:33.576929 6254 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 06:46:33.578004 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.554889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:52726)
0809 06:46:33.555563 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.555583 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.555969 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.556533 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.556538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.557232 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.557596 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.557609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.559082 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.559088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.560791 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.560806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.561925 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.561931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.562127 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.562907 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.562932 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.564340 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.571403 (+ 7063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.571408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.571412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.571905 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.573139 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.573143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.573146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.573352 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.574523 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.574527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.574761 (+ 234us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.576670 (+ 1909us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.577024 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":640,"threads_started":1}
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.254@KRBTEST.COM for oryx/127.27.209.252@KRBTEST.COM
I20260809 06:46:33.593892 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.556760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.27.209.253:33869)
0809 06:46:33.556927 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.556933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.576302 (+ 19369us) server_negotiation.cc:407] Connection header received
0809 06:46:33.576450 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.576455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.576533 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.576627 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.578001 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.579187 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.583688 (+ 4501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.584311 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.584407 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.587321 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.587329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.587334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.587374 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.590680 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.591468 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.591473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.591476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.591664 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.592144 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.592149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.592152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.592513 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.592639 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.593330 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.593558 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 06:46:33.600190 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.556351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:44976)
0809 06:46:33.575984 (+ 19633us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.576002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.576340 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.576800 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.576805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.577549 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.577842 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.577854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.580801 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.580806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.583498 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.583511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.584742 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.584749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.584937 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.585509 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.585536 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.587075 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.590830 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.590835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.590839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.591327 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.591817 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.591822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.591825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.592007 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.592762 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.592766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.592957 (+ 191us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.599337 (+ 6380us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.599709 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19520,"thread_start_us":19325,"threads_started":1}
I20260809 06:46:33.600950 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.583776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36580)
0809 06:46:33.583971 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.583989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.584080 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.585040 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.585045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.585468 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.585643 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.585654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.587948 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.587954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.589311 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.589327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.590391 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.590398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.590565 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.591094 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.591117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.593742 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.598115 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.598120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.598124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.598635 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.599107 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.599113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.599117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.599296 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.600008 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.600013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.600183 (+ 170us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.600326 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.600599 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 06:46:33.601603 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.566436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.27.209.252:36221)
0809 06:46:33.566613 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.566618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.579167 (+ 12549us) server_negotiation.cc:407] Connection header received
0809 06:46:33.579650 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.579656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.579733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.579798 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.581982 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.583059 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.587424 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.588145 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.588248 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.594730 (+ 6482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.594735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.594740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.594771 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.598566 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.599346 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.599351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.599353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.599563 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.600034 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.600039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.600042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.600413 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.600523 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.601039 (+ 516us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.601247 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":47360}
I20260809 06:46:33.602491 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.576732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36570)
0809 06:46:33.579083 (+ 2351us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.579100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.579206 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.580092 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.580096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.580600 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.580859 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.580870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.583310 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.583315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.585680 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.585705 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.588521 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.588526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.588685 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.589948 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.589968 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.594545 (+ 4577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.598716 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.598720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.598725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.599235 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.599711 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.599715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.599717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.599918 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.600599 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.600603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.600754 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.601828 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.602138 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2267,"thread_start_us":3625,"threads_started":1}
I20260809 06:46:33.603030 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.584074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.27.209.252:36221)
0809 06:46:33.584642 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.584647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.584679 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:46:33.584767 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.584772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.584860 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.585784 (+ 924us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.586509 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.587788 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.589495 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.590185 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.590844 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.593988 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.593995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.593999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.594033 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.597958 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.598799 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.598804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.598808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.598961 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.599447 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.599452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.599459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.599858 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.602319 (+ 2461us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.602432 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.602625 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":321,"threads_started":1}
I20260809 06:46:33.604511 5195 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.209.252:36221: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5"
member_type: NON_VOTER
last_known_addr {
host: "127.27.209.252"
port: 36221
}
attrs {
promote: true
}
}
dest_uuid: "70dee95e74b741408369a89d11608f0b"
cas_config_opid_index: 5
I20260809 06:46:33.605611 5195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } }
I20260809 06:46:33.607466 6031 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.608033 6096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 LEARNER]: Refusing update from remote peer 70dee95e74b741408369a89d11608f0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 06:46:33.608116 6210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:33.610512 6031 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.209.252:36221
I20260809 06:46:33.610666 6287 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 06:46:33.611470 6210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Refusing update from remote peer 70dee95e74b741408369a89d11608f0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 06:46:33.627892 6035 master.cc:537] Master@127.27.209.252:36221 shutting down...
I20260809 06:46:33.629629 6289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:33.631822 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.632125 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:33.630870 6096 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.638578 6035 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:33.639017 6035 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:33.639256 6035 tablet_replica.cc:330] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: stopping tablet replica
I20260809 06:46:33.647912 6035 master.cc:552] Master@127.27.209.252:36221 shutdown complete.
I20260809 06:46:33.649307 6035 master_runner.cc:305] Clearing existing system tablet
I20260809 06:46:33.649170 6130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: NON_VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: true } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.649498 6130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: LEARNER
I20260809 06:46:33.650230 6035 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:33.650665 6289 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: attempting to promote NON_VOTER 8094d282293f4843b5d01fb26cb0290b to VOTER
I20260809 06:46:33.651508 6289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } }
I20260809 06:46:33.653759 6096 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 LEARNER]: Refusing update from remote peer 70dee95e74b741408369a89d11608f0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 06:46:33.654253 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:33.654299 6295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 06:46:33.661706 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.661246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36594)
0809 06:46:33.661420 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.661525 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:36221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:46:33.663009 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.662674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36600)
0809 06:46:33.662803 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.662880 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:36221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 06:46:33.663255 5168 consensus_peers.cc:487] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b -> Peer 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252:36221): Couldn't send request to peer 85f7189c52a64f1bb9c06f2bce3196f5. Status: Network error: Client connection negotiation failed: client connection to 127.27.209.252:36221: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 06:46:33.664314 6286 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8094d282293f4843b5d01fb26cb0290b (127.27.209.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.668834 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.669091 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:33.669277 6095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8094d282293f4843b5d01fb26cb0290b (127.27.209.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.670013 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.670295 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:33.673511 6130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.673766 6130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:46:33.675824 6035 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.004s sys 0.004s
I20260809 06:46:33.683480 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:33.683907 6035 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:33.684297 6035 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.005s
I20260809 06:46:33.684438 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
uuid: "85f7189c52a64f1bb9c06f2bce3196f5"
format_stamp: "Formatted at 2026-08-09 06:46:33 on dist-test-slave-jpj1"
server_key: "653ac8b9357a9217e46cf981def772b4"
server_key_iv: "7278620552df2637f03e29949ff36099"
server_key_version: "encryptionkey@0"
I20260809 06:46:33.689632 6035 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:33.696990 6035 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 06:46:33.697173 6035 log.cc:1196] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 06:46:33.697666 6035 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Deleting consensus metadata
I20260809 06:46:33.697926 6035 master_runner.cc:315] Copying system tablet from 127.27.209.254:35661
I20260809 06:46:33.700563 6035 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: tablet copy: Beginning tablet copy session from remote peer at address 127.27.209.254:35661
I20260809 06:46:33.719897 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.701444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.27.209.254:35661)
0809 06:46:33.701672 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:33.701678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:33.705624 (+ 3946us) server_negotiation.cc:407] Connection header received
0809 06:46:33.705838 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:33.705844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:33.705955 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:33.706065 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:33.707125 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.708262 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.710083 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.710746 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.710875 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.713252 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:33.713259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:33.713264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:33.713313 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:33.716882 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.717652 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.717658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.717662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.717821 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:33.718239 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:33.718244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:33.718247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:33.718623 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:33.718768 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:33.719229 (+ 461us) server_negotiation.cc:299] Negotiation successful
0809 06:46:33.719518 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 06:46:33.720131 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:33.701365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:52734)
0809 06:46:33.705483 (+ 4118us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:33.705504 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:33.705673 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:33.706132 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:33.706137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:33.706711 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:33.706968 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.706981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.708454 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.708460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.709922 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:33.709942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:33.710931 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:33.710938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:33.711120 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:33.711887 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:33.711909 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:33.713053 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:33.717006 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.717012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.717016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.717551 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.717933 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:33.717938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:33.717942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:33.718132 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:33.718764 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:33.718770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:33.718970 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 06:46:33.719134 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 06:46:33.719490 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3973,"thread_start_us":3771,"threads_started":1}
I20260809 06:46:33.722533 5224 tablet_copy_service.cc:140] P 70dee95e74b741408369a89d11608f0b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 85f7189c52a64f1bb9c06f2bce3196f5 ({username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:52734)
I20260809 06:46:33.722678 5224 tablet_copy_service.cc:161] P 70dee95e74b741408369a89d11608f0b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 85f7189c52a64f1bb9c06f2bce3196f5 at {username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:52734: session id = 85f7189c52a64f1bb9c06f2bce3196f5-00000000000000000000000000000000
I20260809 06:46:33.725683 5224 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 06:46:33.727684 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 06:46:33.734378 6035 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: tablet copy: Starting download of 0 data blocks...
I20260809 06:46:33.734675 6035 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: tablet copy: Starting download of 1 WAL segments...
I20260809 06:46:33.737380 6035 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 06:46:33.740782 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 5281
I20260809 06:46:33.753048 5224 tablet_copy_service.cc:342] P 70dee95e74b741408369a89d11608f0b: Request end of tablet copy session 85f7189c52a64f1bb9c06f2bce3196f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:52734
I20260809 06:46:33.753228 5224 tablet_copy_service.cc:433] P 70dee95e74b741408369a89d11608f0b: ending tablet copy session 85f7189c52a64f1bb9c06f2bce3196f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 85f7189c52a64f1bb9c06f2bce3196f5
I20260809 06:46:33.765519 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:33.767939 6317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:33.769554 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:34399
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:35661
--webserver_port=46479
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:33.769763 6318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:33.782675 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:33.783301 6035 server_base.cc:1034] running on GCE node
I20260809 06:46:33.783705 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:33.784372 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:33.785705 6035 hybrid_clock.cc:648] HybridClock initialized: now 1786257993785648 us; error 39 us; skew 500 ppm
I20260809 06:46:33.786028 6035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:33 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:33.788273 6035 init.cc:377] Logged in from keytab as oryx/127.27.209.252@KRBTEST.COM (short username oryx)
I20260809 06:46:33.789654 6035 webserver.cc:460] Webserver started at http://127.27.209.252:34987/ using document root <none> and password file <none>
I20260809 06:46:33.790061 6035 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:33.790205 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:33.793252 6035 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 06:46:33.795293 6334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:33.795746 6035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 06:46:33.796048 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
uuid: "85f7189c52a64f1bb9c06f2bce3196f5"
format_stamp: "Formatted at 2026-08-09 06:46:33 on dist-test-slave-jpj1"
server_key: "653ac8b9357a9217e46cf981def772b4"
server_key_iv: "7278620552df2637f03e29949ff36099"
server_key_version: "encryptionkey@0"
I20260809 06:46:33.796337 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:33.812142 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:33.816512 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:33.847720 6035 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.252:36221
I20260809 06:46:33.847801 6413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.252:36221 every 8 connection(s)
I20260809 06:46:33.848976 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
I20260809 06:46:33.855010 6414 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:46:33.858879 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Bootstrap starting.
I20260809 06:46:33.878127 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:46:33.878656 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Bootstrap complete.
I20260809 06:46:33.879774 6414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } }
I20260809 06:46:33.880069 6414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85f7189c52a64f1bb9c06f2bce3196f5, State: Initialized, Role: LEARNER
I20260809 06:46:33.880347 6414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } }
I20260809 06:46:33.881493 6414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:33.881244 6417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:33.881856 6417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: This master's current role is: LEARNER
I20260809 06:46:33.890204 6432 catalog_manager.cc:1259] Loaded cluster ID: dd41263487264d9f942755b4bead47d3
I20260809 06:46:33.890286 6432 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: loading cluster ID for follower catalog manager: success
I20260809 06:46:33.892743 6432 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: acquiring CA information for follower catalog manager: success
I20260809 06:46:33.894582 6432 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:46:34.106071 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.100045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36614)
0809 06:46:34.100334 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.100356 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.100502 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.101277 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.101283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.101314 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:34.101472 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.101481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.102740 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.102746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.104035 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.104049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.105235 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.105241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.105457 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.105493 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.105579 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.105718 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260809 06:46:34.106079 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.100157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.27.209.252:36221)
0809 06:46:34.100961 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.100965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.100981 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:46:34.101039 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.101043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.101110 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.101216 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:34.101609 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.102582 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.104190 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.105065 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.105248 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.105443 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.105613 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.105744 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":544,"threads_started":1}
I20260809 06:46:34.106977 6374 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 06:46:34.107573 6417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70dee95e74b741408369a89d11608f0b. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:34.107858 6417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: This master's current role is: LEARNER
W20260809 06:46:34.172674 6325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:34.172966 6325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:34.173071 6325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:34.180399 6325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:34.180512 6325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:34.180639 6325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:34.190994 6325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:34399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=46479
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:34.192342 6325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:34.193672 6325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:34.206751 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:34.206692 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:34.206776 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:34.207804 6325 server_base.cc:1034] running on GCE node
I20260809 06:46:34.208410 6325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:34.209304 6325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:34.210498 6325 hybrid_clock.cc:648] HybridClock initialized: now 1786257994210445 us; error 45 us; skew 500 ppm
I20260809 06:46:34.210793 6325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:34.213795 6325 init.cc:377] Logged in from keytab as oryx/127.27.209.193@KRBTEST.COM (short username oryx)
I20260809 06:46:34.215178 6325 webserver.cc:460] Webserver started at http://127.27.209.193:46479/ using document root <none> and password file <none>
I20260809 06:46:34.215888 6325 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:34.216028 6325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:34.221333 6325 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:46:34.224028 6464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:34.225407 6325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:46:34.225551 6325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "22ddf774e2214df7a6f8e0c3ebca0377"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "76bf7b5db2520f33cf9bd0fee83216af"
server_key_iv: "e9d9eb93597df3498921b98657016a9d"
server_key_version: "encryptionkey@0"
I20260809 06:46:34.226498 6325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:34.263139 6325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:34.267673 6325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:34.267928 6325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:34.268908 6325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:34.270289 6325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:34.270390 6325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:34.270476 6325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:34.270538 6325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:34.305727 6325 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:34399
I20260809 06:46:34.305794 6643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:34399 every 8 connection(s)
I20260809 06:46:34.307176 6325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:34.311806 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 6325
I20260809 06:46:34.312182 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 5486
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:34.347795 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.310555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:41625 (local address 127.27.209.253:33869)
0809 06:46:34.320137 (+ 9582us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.320144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.324546 (+ 4402us) server_negotiation.cc:407] Connection header received
0809 06:46:34.325395 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.325402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.325512 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.325851 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.328482 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.329770 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.332406 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.333125 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.333228 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.339848 (+ 6620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.339855 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.339860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.339900 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.343428 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.344427 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.344432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.344436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.344589 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.345444 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.345450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.345453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.345795 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.345945 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.347136 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.347400 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9457,"thread_start_us":9389,"threads_started":1}
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.252@KRBTEST.COM
I20260809 06:46:34.349665 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.310921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.27.209.193:41625)
0809 06:46:34.321856 (+ 10935us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.321917 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.324931 (+ 3014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.326345 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.326359 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.327281 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:34.328268 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.328294 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.330010 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.330016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.332237 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.332253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.333617 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.333626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.333998 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.334905 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:34.334940 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:34.338560 (+ 3620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:34.343627 (+ 5067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.343636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.343653 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.344266 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.344718 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.344722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.344725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.345290 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.346029 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:34.346037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:34.346679 (+ 642us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.347005 (+ 326us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.348119 (+ 1114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9607,"thread_start_us":28338,"threads_started":1}
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.194@KRBTEST.COM
I20260809 06:46:34.352324 6647 heartbeater.cc:347] Connected to a master server at 127.27.209.253:33869
I20260809 06:46:34.352684 6647 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:34.353431 6647 heartbeater.cc:511] Master 127.27.209.253:33869 requested a full tablet report, sending...
I20260809 06:46:34.355862 6083 ts_manager.cc:194] Registered new tserver with Master: 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193:34399)
I20260809 06:46:34.357605 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.310626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:56785 (local address 127.27.209.254:35661)
0809 06:46:34.311706 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.311713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.337498 (+ 25785us) server_negotiation.cc:407] Connection header received
0809 06:46:34.337663 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.337669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.337753 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.338013 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.339662 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.340746 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.343234 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.343933 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.344038 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.350568 (+ 6530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.350575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.350579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.350614 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.354375 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.355265 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.355271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.355274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.355439 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.355927 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.355932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.355935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.356290 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.356437 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.356997 (+ 560us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.357229 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":819,"threads_started":1}
I20260809 06:46:34.358280 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.310920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:55835 (local address 127.27.209.252:36221)
0809 06:46:34.311125 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.311131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.339946 (+ 28815us) server_negotiation.cc:407] Connection header received
0809 06:46:34.340209 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.340215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.340296 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.340371 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.341806 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.342890 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.344750 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.345347 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.345445 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.349846 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.349851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.349855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.349886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.355609 (+ 5723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.356402 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.356407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.356410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.356569 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.357045 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.357049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.357052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.357342 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.357440 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.357870 (+ 430us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.358052 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:46:34.358594 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.310921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:56785)
0809 06:46:34.337186 (+ 26265us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.337208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.337542 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.338240 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.338244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.338987 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:34.339334 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.339349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.341597 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.341602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.343071 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.343082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.344349 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.344355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.344609 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.345171 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:34.345195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:34.350363 (+ 5168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:34.354532 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.354537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.354540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.355115 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.355621 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.355627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.355630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.355782 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.356528 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:34.356537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:34.356697 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.357884 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.358175 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26087,"mutex_wait_us":99,"spinlock_wait_cycles":49536,"thread_start_us":9608,"threads_started":1}
I20260809 06:46:34.359143 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.313531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.27.209.193:55835)
0809 06:46:34.339667 (+ 26136us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.339687 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.340050 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.340569 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.340573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.341109 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:34.341449 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.341462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.343071 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.343075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.344586 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.344599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.345723 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.345729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.345929 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.347285 (+ 1356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:34.347307 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:34.349660 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:34.355767 (+ 6107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.355772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.355776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.356266 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.356711 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.356716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.356719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.356915 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.357503 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:34.357506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:34.357612 (+ 106us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.358512 (+ 900us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.358798 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25986,"mutex_wait_us":3,"thread_start_us":23257,"threads_started":1}
I20260809 06:46:34.359609 6644 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35661
I20260809 06:46:34.359753 6644 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:34.360219 6644 heartbeater.cc:511] Master 127.27.209.254:35661 requested a full tablet report, sending...
I20260809 06:46:34.360293 6646 heartbeater.cc:347] Connected to a master server at 127.27.209.252:36221
I20260809 06:46:34.360407 6646 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:34.360654 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.313127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.27.209.194:45695)
0809 06:46:34.332154 (+ 19027us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.332163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.339515 (+ 7352us) server_negotiation.cc:407] Connection header received
0809 06:46:34.340422 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.340433 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.340720 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.340958 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.342952 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.344518 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.346283 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.346914 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.347022 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.352911 (+ 5889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.352926 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.352942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.352985 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.356714 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.357612 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.357621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.357624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.357799 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.358232 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.358237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.358241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.358667 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.358966 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.360086 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.360344 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18828,"thread_start_us":19040,"threads_started":1}
I20260809 06:46:34.360816 6646 heartbeater.cc:511] Master 127.27.209.252:36221 requested a full tablet report, sending...
I20260809 06:46:34.361438 5195 ts_manager.cc:194] Re-registered known tserver with Master: 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193:34399)
I20260809 06:46:34.362183 5665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:34.362301 5195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:56785
I20260809 06:46:34.362397 6358 ts_manager.cc:194] Registered new tserver with Master: 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193:34399)
I20260809 06:46:34.444896 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 5486
I20260809 06:46:34.468014 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:45695
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:35661
--webserver_port=44455
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:46:34.638540 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: NON_VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: true } } }
I20260809 06:46:34.638918 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:34.640292 6668 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: attempting to promote NON_VOTER 85f7189c52a64f1bb9c06f2bce3196f5 to VOTER
I20260809 06:46:34.640877 6668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } }
I20260809 06:46:34.642694 6095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Refusing update from remote peer 70dee95e74b741408369a89d11608f0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 06:46:34.642742 6374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 LEARNER]: Refusing update from remote peer 70dee95e74b741408369a89d11608f0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 06:46:34.643311 6295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 06:46:34.643719 6667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 06:46:34.645954 6295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.648618 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.648857 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:34.649356 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.649619 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:34.649704 6095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.651569 6667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.651811 6667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:34.652784 6374 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.659163 6673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.659673 6673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:46:34.659952 6674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70dee95e74b741408369a89d11608f0b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:34.660254 6674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 06:46:34.799852 6666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:34.800125 6666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:34.800262 6666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:34.807403 6666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:34.807551 6666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:34.807690 6666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:34.817678 6666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:45695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=44455
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:34.818986 6666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:34.820385 6666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:34.833209 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:34.833246 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:34.833276 6687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:34.834318 6666 server_base.cc:1034] running on GCE node
I20260809 06:46:34.834918 6666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:34.835819 6666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:34.837025 6666 hybrid_clock.cc:648] HybridClock initialized: now 1786257994836971 us; error 46 us; skew 500 ppm
I20260809 06:46:34.837316 6666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:34.840289 6666 init.cc:377] Logged in from keytab as oryx/127.27.209.194@KRBTEST.COM (short username oryx)
I20260809 06:46:34.841612 6666 webserver.cc:460] Webserver started at http://127.27.209.194:44455/ using document root <none> and password file <none>
I20260809 06:46:34.842265 6666 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:34.842402 6666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:34.847581 6666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:34.850420 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:34.851867 6666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:46:34.852021 6666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "228c3285dfbd49eaa4a9978cdc0464db"
format_stamp: "Formatted at 2026-08-09 06:46:31 on dist-test-slave-jpj1"
server_key: "0b9fae75409b10b1ae65ebea0f678fe9"
server_key_iv: "a4d872ef2c9dcf63dbe4967c34ee89c2"
server_key_version: "encryptionkey@0"
I20260809 06:46:34.853008 6666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:34.868263 6666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:34.871925 6666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:34.872184 6666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:34.873080 6666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:34.874361 6666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:34.874454 6666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:34.874557 6666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:34.874629 6666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:34.904666 6666 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:45695
I20260809 06:46:34.904726 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:45695 every 8 connection(s)
I20260809 06:46:34.906199 6666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:34.909408 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 6666
I20260809 06:46:34.909722 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 5680
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:34.934382 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.910372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:51385)
0809 06:46:34.913748 (+ 3376us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.913766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.913963 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.914593 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.914604 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.915479 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:34.916351 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.916377 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.917908 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.917914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.919861 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.919877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.920893 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.920900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.921196 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.921972 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:34.922001 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:34.925112 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:34.929110 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.929119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.929137 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.929829 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.930302 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.930307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.930311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.930585 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.931278 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:34.931286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:34.931719 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.932063 (+ 344us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.932603 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3260,"thread_start_us":14776,"threads_started":1}
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.252@KRBTEST.COM
I20260809 06:46:34.935755 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.911065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:51385 (local address 127.27.209.254:35661)
0809 06:46:34.911510 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.911516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.913873 (+ 2357us) server_negotiation.cc:407] Connection header received
0809 06:46:34.914119 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.914125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.914214 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.914331 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.916524 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.917744 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.919995 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.920664 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.920791 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.925390 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.925396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.925401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.925449 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.928948 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.929994 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.930000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.930003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.930171 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.930716 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.930721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.930724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.931059 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.931198 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.935124 (+ 3926us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.935365 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":516,"threads_started":1}
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257994, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
Aug 09 06:46:34 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.195@KRBTEST.COM
I20260809 06:46:34.938223 6874 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35661
I20260809 06:46:34.938575 6874 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:34.939306 6874 heartbeater.cc:511] Master 127.27.209.254:35661 requested a full tablet report, sending...
I20260809 06:46:34.940937 5195 ts_manager.cc:194] Re-registered known tserver with Master: 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:34.942098 5195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:51385
I20260809 06:46:34.944689 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.909860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.27.209.194:55985)
0809 06:46:34.927426 (+ 17566us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.927445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.927676 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.928236 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.928241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.928801 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:34.929101 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.929113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.930675 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.930680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.932313 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.932328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.933348 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.933355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.933541 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.935165 (+ 1624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:34.935185 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:34.937679 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:34.941742 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.941747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.941750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.942261 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.942696 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.942702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.942706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.942908 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.943649 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:34.943654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:34.943865 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.944009 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.944278 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17391,"mutex_wait_us":3215,"thread_start_us":769,"threads_started":1}
I20260809 06:46:34.945232 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.909860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.27.209.194:43891)
0809 06:46:34.910793 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:34.910843 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:34.913295 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:34.927073 (+ 13778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:34.927078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:34.927570 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:34.927908 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.927972 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.929543 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.929548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.931240 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:34.931277 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.932593 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.932600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:34.932765 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.933856 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:34.933876 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:34.936973 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:34.942443 (+ 5470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.942448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.942452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.942931 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.943379 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:34.943384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:34.943386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:34.943573 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:34.944259 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:34.944263 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:34.944389 (+ 126us) client_negotiation.cc:769] Sending connection context
0809 06:46:34.944518 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 06:46:34.944840 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260809 06:46:34.945544 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.909652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:55985 (local address 127.27.209.253:33869)
0809 06:46:34.926389 (+ 16737us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.926395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.927585 (+ 1190us) server_negotiation.cc:407] Connection header received
0809 06:46:34.927843 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.927848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.927946 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.928051 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.929292 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.930489 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.932482 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.933141 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.933259 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.937891 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.937897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.937901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.937952 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.941587 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.942406 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.942412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.942415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.942562 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.943045 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.943050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.943053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.943384 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.943585 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.944991 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.945211 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16609,"thread_start_us":16515,"threads_started":1}
I20260809 06:46:34.946148 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.909333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:43891 (local address 127.27.209.252:36221)
0809 06:46:34.926762 (+ 17429us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.926767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.926792 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:46:34.926841 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.926845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.926919 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.927014 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.928152 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.929371 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.931474 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.932194 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.932321 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.937197 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.937203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.937206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.937254 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.942283 (+ 5029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.943083 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.943088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.943091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.943239 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.943711 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.943716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.943719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.944055 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.944185 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.945604 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.945830 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17320,"thread_start_us":17185,"threads_started":1}
I20260809 06:46:34.946278 6877 heartbeater.cc:347] Connected to a master server at 127.27.209.253:33869
I20260809 06:46:34.946403 6877 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:34.946831 6877 heartbeater.cc:511] Master 127.27.209.253:33869 requested a full tablet report, sending...
I20260809 06:46:34.946869 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:34.911421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39356 (local address 127.27.209.195:43853)
0809 06:46:34.924133 (+ 12712us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:34.924141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:34.925372 (+ 1231us) server_negotiation.cc:407] Connection header received
0809 06:46:34.925450 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:34.925459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:34.925714 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:34.925904 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:34.928189 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.929841 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.931868 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:34.932757 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:34.932875 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:34.938776 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:34.938791 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:34.938810 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:34.938860 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:34.942769 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.943625 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.943633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.943636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.943806 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:34.944266 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:34.944270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:34.944273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:34.944681 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:34.944961 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:34.946241 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0809 06:46:34.946510 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12485,"thread_start_us":12308,"threads_started":1}
I20260809 06:46:34.947263 6875 heartbeater.cc:347] Connected to a master server at 127.27.209.252:36221
I20260809 06:46:34.947387 6875 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:34.947686 6875 heartbeater.cc:511] Master 127.27.209.252:36221 requested a full tablet report, sending...
I20260809 06:46:34.948154 5867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:34.948889 6083 ts_manager.cc:194] Registered new tserver with Master: 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:34.949106 6358 ts_manager.cc:194] Registered new tserver with Master: 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:35.030164 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 5680
I20260809 06:46:35.051791 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:43853
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:35661
--webserver_port=32887
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:46:35.365054 6644 heartbeater.cc:503] Master 127.27.209.254:35661 was elected leader, sending a full tablet report...
W20260809 06:46:35.383994 6901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:35.384302 6901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:35.384462 6901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:35.391590 6901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:35.391711 6901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:35.391875 6901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:35.401800 6901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:43853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=32887
--tserver_master_addrs=127.27.209.254:35661,127.27.209.253:33869,127.27.209.252:36221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:35.403599 6901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:35.405236 6901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:35.419112 6916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:35.419152 6912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:35.419173 6909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:35.419781 6901 server_base.cc:1034] running on GCE node
I20260809 06:46:35.420358 6901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:35.421237 6901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:35.422422 6901 hybrid_clock.cc:648] HybridClock initialized: now 1786257995422380 us; error 32 us; skew 500 ppm
I20260809 06:46:35.422701 6901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:35 dist-test-slave-jpj1 krb5kdc[5129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257995, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:35.425793 6901 init.cc:377] Logged in from keytab as oryx/127.27.209.195@KRBTEST.COM (short username oryx)
I20260809 06:46:35.427206 6901 webserver.cc:460] Webserver started at http://127.27.209.195:32887/ using document root <none> and password file <none>
I20260809 06:46:35.427912 6901 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:35.428047 6901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:35.433416 6901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:46:35.436172 6926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:35.437518 6901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:35.437695 6901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "bbdce250244643f4968b48eab45d191c"
format_stamp: "Formatted at 2026-08-09 06:46:32 on dist-test-slave-jpj1"
server_key: "c2abcfa4b650a3753222f3cd667b5234"
server_key_iv: "9f064f80e10ffe347ef22670f4113848"
server_key_version: "encryptionkey@0"
I20260809 06:46:35.438651 6901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:35.465020 6901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:35.468704 6901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:35.468968 6901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:35.469903 6901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:35.471223 6901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:35.471308 6901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:35.471408 6901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:35.471481 6901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:35.501646 6901 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:43853
I20260809 06:46:35.501711 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:43853 every 8 connection(s)
I20260809 06:46:35.503053 6901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:35.513355 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 6901
Aug 09 06:46:35 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257995, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.252@KRBTEST.COM
Aug 09 06:46:35 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257995, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.253@KRBTEST.COM
I20260809 06:46:35.550863 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.507092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:52499 (local address 127.27.209.252:36221)
0809 06:46:35.518972 (+ 11880us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.518979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.519005 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:46:35.521304 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.521310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.521410 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.521490 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.525230 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.526315 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.528161 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.528729 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.528838 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.533246 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.533254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.533259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.533299 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.537106 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.538407 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.538413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.538416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.538572 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.539023 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.539027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.539030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.539318 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.539422 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.549956 (+ 10534us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.550219 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11745,"thread_start_us":11632,"threads_started":1}
I20260809 06:46:35.553833 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.506474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.27.209.195:52499)
0809 06:46:35.516293 (+ 9819us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.516307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.521144 (+ 4837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.521685 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.521697 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.522811 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:35.523654 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.523683 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.526502 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.526508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.528001 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.528016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.529117 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.529123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.529614 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.530221 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:35.530249 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:35.532973 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:35.537808 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:35.537812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:35.537815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:35.538255 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:35.538719 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:35.538724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:35.538728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:35.538902 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:35.539486 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:35.539489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:35.549490 (+ 10001us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.551472 (+ 1982us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.552173 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8413,"mutex_wait_us":9675,"thread_start_us":12585,"threads_started":1}
I20260809 06:46:35.556077 7101 heartbeater.cc:347] Connected to a master server at 127.27.209.252:36221
I20260809 06:46:35.556479 7101 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:35.557440 7101 heartbeater.cc:511] Master 127.27.209.252:36221 requested a full tablet report, sending...
I20260809 06:46:35.557606 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.505891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:40529 (local address 127.27.209.253:33869)
0809 06:46:35.519772 (+ 13881us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.519778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.519807 (+ 29us) server_negotiation.cc:407] Connection header received
0809 06:46:35.521046 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.521052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.521127 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.521208 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.523866 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.525022 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.527435 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.528122 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.528209 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.533648 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.533655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.533659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.533701 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.537106 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.537815 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.537821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.537824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.537970 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.538451 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.538456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.538459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.538841 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.538965 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.556887 (+ 17922us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.557163 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13766,"thread_start_us":13651,"threads_started":1}
I20260809 06:46:35.558791 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.507508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.27.209.195:40529)
0809 06:46:35.516113 (+ 8605us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.516171 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.520344 (+ 4173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.521685 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.521697 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.522580 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:35.523654 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.523683 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.525225 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.525230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.527260 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.527275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.528539 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.528546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.528831 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.529601 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:35.529633 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:35.533416 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:35.537258 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:35.537266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:35.537279 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:35.537685 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:35.538102 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:35.538107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:35.538110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:35.538334 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:35.539048 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:35.539056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:35.556519 (+ 17463us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.558013 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.558354 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7931,"mutex_wait_us":10101,"thread_start_us":7243,"threads_started":1}
I20260809 06:46:35.559491 6358 ts_manager.cc:194] Registered new tserver with Master: bbdce250244643f4968b48eab45d191c (127.27.209.195:43853)
I20260809 06:46:35.560524 7102 heartbeater.cc:347] Connected to a master server at 127.27.209.253:33869
I20260809 06:46:35.560655 7102 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:35.561156 7102 heartbeater.cc:511] Master 127.27.209.253:33869 requested a full tablet report, sending...
I20260809 06:46:35.562395 6083 ts_manager.cc:194] Registered new tserver with Master: bbdce250244643f4968b48eab45d191c (127.27.209.195:43853)
Aug 09 06:46:35 dist-test-slave-jpj1 krb5kdc[5129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257995, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:35.572731 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.505815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:50705 (local address 127.27.209.254:35661)
0809 06:46:35.518224 (+ 12409us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.518232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.520459 (+ 2227us) server_negotiation.cc:407] Connection header received
0809 06:46:35.520804 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.520810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.520897 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.520988 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.523881 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.525025 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.549082 (+ 24057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.549713 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.549861 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.564035 (+ 14174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.564043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.564047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.564093 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.567730 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.568529 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.568535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.568539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.568698 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.570966 (+ 2268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.570971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.570975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.571328 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.571469 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.572114 (+ 645us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.572356 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12259,"thread_start_us":12407,"threads_started":1}
I20260809 06:46:35.573675 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.507977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.195:50705)
0809 06:46:35.520349 (+ 12372us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.520369 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.520498 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.521688 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.521698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.522798 (+ 1100us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:35.523672 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.523688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.530083 (+ 6395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.530089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.548912 (+ 18823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.548931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.559033 (+ 10102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.559040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.559308 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.560884 (+ 1576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:35.560913 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:35.563750 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:35.567888 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:35.567893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:35.567897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:35.568391 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:35.568848 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:35.568853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:35.568857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:35.569015 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:35.571665 (+ 2650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:35.571671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:35.571830 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.572976 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.573275 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12235,"mutex_wait_us":38,"thread_start_us":7683,"threads_started":1}
I20260809 06:46:35.574865 7099 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35661
I20260809 06:46:35.575008 7099 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:35.575397 7099 heartbeater.cc:511] Master 127.27.209.254:35661 requested a full tablet report, sending...
I20260809 06:46:35.576762 5195 ts_manager.cc:194] Re-registered known tserver with Master: bbdce250244643f4968b48eab45d191c (127.27.209.195:43853)
I20260809 06:46:35.577822 5195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:50705
I20260809 06:46:35.580543 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.530682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.27.209.252:36221)
0809 06:46:35.551133 (+ 20451us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.551139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.560746 (+ 9607us) server_negotiation.cc:407] Connection header received
0809 06:46:35.563570 (+ 2824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.563576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.563672 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.563744 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.566207 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.567291 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.569155 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.569755 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.569868 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.573957 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.573964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.573968 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.574006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.577269 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.578146 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.578152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.578154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.578316 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.578779 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.578794 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.578798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.579122 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.579379 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.580001 (+ 622us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.580224 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20332,"thread_start_us":29893,"threads_started":1}
I20260809 06:46:35.585592 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.534895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.27.209.253:33869)
0809 06:46:35.557845 (+ 22950us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.557852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.569840 (+ 11988us) server_negotiation.cc:407] Connection header received
0809 06:46:35.569997 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.570002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.570100 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.570197 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.572538 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.573366 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.574903 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.575517 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.575665 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.579027 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.579034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.579039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.579079 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.582762 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.583431 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.583436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.583439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.583602 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.584041 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.584046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.584050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.584402 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.584645 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.585047 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.585248 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22846,"thread_start_us":25757,"threads_started":1}
I20260809 06:46:35.588091 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.524091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.27.209.254:35661)
0809 06:46:35.570357 (+ 46266us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.570363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.570380 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:46:35.570425 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.570429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.570496 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.570849 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.574260 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.575366 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.578157 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.579142 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.580662 (+ 1520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.582493 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.582497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.582500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.582527 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.585418 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.586116 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.586120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.586122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.586241 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.586607 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.586612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.586615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.586972 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.587218 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.587512 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.587764 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46132,"thread_start_us":46002,"threads_started":1}
I20260809 06:46:35.591724 5195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:35.598543 5195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:35.631865 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.616815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:43853 (local address 127.0.0.1:39358)
0809 06:46:35.620450 (+ 3635us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.620464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.620822 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.622027 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.622031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.622056 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:35.622461 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.622473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.624772 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.624778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.627094 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.627111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.630301 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.630307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.630467 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.631231 (+ 764us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.631315 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.631452 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3539,"thread_start_us":1499,"threads_started":1}
I20260809 06:46:35.633224 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.617518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.27.209.195:43853)
0809 06:46:35.620057 (+ 2539us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.620064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.620697 (+ 633us) server_negotiation.cc:407] Connection header received
0809 06:46:35.621021 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.621031 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.621332 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.621651 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:35.622709 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.624566 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.628954 (+ 4388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.630134 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.630295 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.630481 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.632854 (+ 2373us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.632991 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2396,"thread_start_us":2206,"threads_started":1}
I20260809 06:46:35.635186 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.616387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:34399 (local address 127.0.0.1:41704)
0809 06:46:35.618620 (+ 2233us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.618638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.618956 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.628553 (+ 9597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.628556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.628572 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:35.628921 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.628937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.630815 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.630820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.632674 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.632687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.634430 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.634436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.634613 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.634645 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.634749 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.634872 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2074,"thread_start_us":3811,"threads_started":1}
I20260809 06:46:35.635336 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.616652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.27.209.193:34399)
0809 06:46:35.627719 (+ 11067us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.627729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.627762 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:46:35.627881 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.627889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.628173 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.628598 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:35.629111 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.630692 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.632828 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.634214 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.634383 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.634557 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.634820 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.635003 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10845,"thread_start_us":10714,"threads_started":1}
I20260809 06:46:35.637485 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 96984501ccfd4cd2babdbfd2d3e89e9d (DEFAULT_TABLE table=test-table [id=236aae700a9646f2979de9b7fbd5cb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:35.639149 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96984501ccfd4cd2babdbfd2d3e89e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:35.639350 6544 tablet_service.cc:1467] Processing CreateTablet for tablet 96984501ccfd4cd2babdbfd2d3e89e9d (DEFAULT_TABLE table=test-table [id=236aae700a9646f2979de9b7fbd5cb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:35.640972 6544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96984501ccfd4cd2babdbfd2d3e89e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:35.646786 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.620805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:45695 (local address 127.0.0.1:55086)
0809 06:46:35.628034 (+ 7229us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.628049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.628285 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.639658 (+ 11373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.639663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.639691 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:35.640043 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.640056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.642079 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.642085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.644076 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.644092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.645795 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.645801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.645978 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.646013 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.646107 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.646422 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7120,"thread_start_us":6983,"threads_started":1}
I20260809 06:46:35.647677 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.620835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.27.209.194:45695)
0809 06:46:35.638560 (+ 17725us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.638568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.638611 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:46:35.638723 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.638731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.638970 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.640264 (+ 1294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:35.640482 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.641944 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.644247 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.645567 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.645742 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.645903 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.647214 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.647387 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17559,"thread_start_us":18512,"threads_started":1}
I20260809 06:46:35.651413 6773 tablet_service.cc:1467] Processing CreateTablet for tablet 96984501ccfd4cd2babdbfd2d3e89e9d (DEFAULT_TABLE table=test-table [id=236aae700a9646f2979de9b7fbd5cb66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:35.652807 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96984501ccfd4cd2babdbfd2d3e89e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:35.655053 7147 tablet_bootstrap.cc:492] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Bootstrap starting.
I20260809 06:46:35.658411 7146 tablet_bootstrap.cc:492] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Bootstrap starting.
I20260809 06:46:35.658852 7147 tablet_bootstrap.cc:654] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:35.660233 7147 log.cc:824] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:35.661798 7146 tablet_bootstrap.cc:654] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:35.663007 7146 log.cc:824] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:35.663362 7147 tablet_bootstrap.cc:492] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: No bootstrap required, opened a new log
I20260809 06:46:35.663679 7147 ts_tablet_manager.cc:1397] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 06:46:35.666193 7146 tablet_bootstrap.cc:492] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: No bootstrap required, opened a new log
I20260809 06:46:35.666460 7146 ts_tablet_manager.cc:1397] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260809 06:46:35.673638 7147 raft_consensus.cc:348] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.674124 7147 raft_consensus.cc:374] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:35.674226 7147 raft_consensus.cc:729] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ddf774e2214df7a6f8e0c3ebca0377, State: Initialized, Role: FOLLOWER
I20260809 06:46:35.674968 7147 consensus_queue.cc:260] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.676605 7146 raft_consensus.cc:348] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.677052 7146 raft_consensus.cc:374] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:35.677150 7146 raft_consensus.cc:729] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdce250244643f4968b48eab45d191c, State: Initialized, Role: FOLLOWER
I20260809 06:46:35.677851 7146 consensus_queue.cc:260] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.680086 7152 tablet_bootstrap.cc:492] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Bootstrap starting.
I20260809 06:46:35.683223 7147 ts_tablet_manager.cc:1428] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260809 06:46:35.684281 7152 tablet_bootstrap.cc:654] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:35.684537 7146 ts_tablet_manager.cc:1428] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260809 06:46:35.685252 7099 heartbeater.cc:503] Master 127.27.209.254:35661 was elected leader, sending a full tablet report...
I20260809 06:46:35.689360 7152 log.cc:824] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:35.693008 7152 tablet_bootstrap.cc:492] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: No bootstrap required, opened a new log
I20260809 06:46:35.693285 7152 ts_tablet_manager.cc:1397] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260809 06:46:35.700358 7152 raft_consensus.cc:348] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.700722 7152 raft_consensus.cc:374] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:35.700816 7152 raft_consensus.cc:729] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228c3285dfbd49eaa4a9978cdc0464db, State: Initialized, Role: FOLLOWER
I20260809 06:46:35.701375 7152 consensus_queue.cc:260] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.702363 6874 heartbeater.cc:503] Master 127.27.209.254:35661 was elected leader, sending a full tablet report...
I20260809 06:46:35.704852 7152 ts_tablet_manager.cc:1428] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
W20260809 06:46:35.762674 7103 tablet.cc:2369] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:35.765172 7155 raft_consensus.cc:482] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:35.765458 7155 raft_consensus.cc:504] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.767231 7155 leader_election.cc:288] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbdce250244643f4968b48eab45d191c (127.27.209.195:43853), 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:35.773566 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.767633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:51079 (local address 127.27.209.195:43853)
0809 06:46:35.767878 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.767885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.768576 (+ 691us) server_negotiation.cc:407] Connection header received
0809 06:46:35.768755 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.768760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.768847 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.768948 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:35.769436 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.770205 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.771963 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.772724 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.772871 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.772947 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.773205 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.773336 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 06:46:35.773813 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.767444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:43853 (local address 127.27.209.193:51079)
0809 06:46:35.768325 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.768340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.768612 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.768999 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.769003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.769022 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:35.769303 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.769311 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.770339 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.770343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.771821 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.771833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.772923 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.772929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.773058 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.773081 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.773159 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.773286 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":853,"threads_started":1}
I20260809 06:46:35.773917 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.767675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:33761 (local address 127.27.209.194:45695)
0809 06:46:35.768066 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.768072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.768938 (+ 866us) server_negotiation.cc:407] Connection header received
0809 06:46:35.769186 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.769190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.769256 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.769328 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:35.769838 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.770564 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.772317 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.773056 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.773216 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.773288 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.773551 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.773670 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260809 06:46:35.774009 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.767488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:45695 (local address 127.27.209.193:33761)
0809 06:46:35.768638 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:35.768652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:35.768998 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:35.769361 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:35.769365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:35.769388 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:35.769666 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.769675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.770712 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.770716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.772159 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:35.772172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.773267 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.773274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:35.773407 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.773431 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:46:35.773509 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:46:35.773628 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":501,"threads_started":1}
I20260809 06:46:35.774873 7030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96984501ccfd4cd2babdbfd2d3e89e9d" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdce250244643f4968b48eab45d191c" is_pre_election: true
I20260809 06:46:35.774885 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96984501ccfd4cd2babdbfd2d3e89e9d" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228c3285dfbd49eaa4a9978cdc0464db" is_pre_election: true
I20260809 06:46:35.775358 6803 raft_consensus.cc:2454] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 0.
I20260809 06:46:35.775350 7030 raft_consensus.cc:2454] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 0.
I20260809 06:46:35.775995 6467 leader_election.cc:302] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228c3285dfbd49eaa4a9978cdc0464db, 22ddf774e2214df7a6f8e0c3ebca0377; no voters:
I20260809 06:46:35.776358 7155 raft_consensus.cc:2781] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:35.776517 7155 raft_consensus.cc:482] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:35.776619 7155 raft_consensus.cc:3037] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:35.778729 7155 raft_consensus.cc:504] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.779253 7155 leader_election.cc:288] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 election: Requested vote from peers bbdce250244643f4968b48eab45d191c (127.27.209.195:43853), 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:35.779718 7030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96984501ccfd4cd2babdbfd2d3e89e9d" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdce250244643f4968b48eab45d191c"
I20260809 06:46:35.779721 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96984501ccfd4cd2babdbfd2d3e89e9d" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228c3285dfbd49eaa4a9978cdc0464db"
I20260809 06:46:35.779892 7030 raft_consensus.cc:3037] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:35.779923 6803 raft_consensus.cc:3037] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:35.782356 7030 raft_consensus.cc:2454] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 1.
I20260809 06:46:35.782380 6803 raft_consensus.cc:2454] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 1.
I20260809 06:46:35.782723 6469 leader_election.cc:302] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ddf774e2214df7a6f8e0c3ebca0377, bbdce250244643f4968b48eab45d191c; no voters:
I20260809 06:46:35.783066 7155 raft_consensus.cc:2781] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:35.783468 7155 raft_consensus.cc:686] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 LEADER]: Becoming Leader. State: Replica: 22ddf774e2214df7a6f8e0c3ebca0377, State: Running, Role: LEADER
I20260809 06:46:35.784070 7155 consensus_queue.cc:237] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:35.789767 5195 catalog_manager.cc:5495] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } health_report { overall_health: UNKNOWN } } }
I20260809 06:46:35.828953 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.823987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.27.209.193:34399)
0809 06:46:35.824166 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.824170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.824189 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:46:35.824292 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.824296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.824374 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.824498 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:35.824811 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.825591 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.826971 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.827375 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.827467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.827943 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:35.828473 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:35.828591 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.828626 (+ 35us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.828692 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:46:35.837103 6803 raft_consensus.cc:1261] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Refusing update from remote peer 22ddf774e2214df7a6f8e0c3ebca0377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:35.837107 7030 raft_consensus.cc:1261] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Refusing update from remote peer 22ddf774e2214df7a6f8e0c3ebca0377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:35.838191 7155 consensus_queue.cc:1035] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:35.838502 7167 consensus_queue.cc:1035] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:46:35.841351 6648 tablet.cc:2369] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:35.853746 7171 mvcc.cc:205] Tried to move back new op lower bound from 7316512750938812416 to 7316512750735695872. Current Snapshot: MvccSnapshot[applied={T|T < 7316512750938812416}]
I20260809 06:46:35.868218 7173 mvcc.cc:205] Tried to move back new op lower bound from 7316512750938812416 to 7316512750735695872. Current Snapshot: MvccSnapshot[applied={T|T < 7316512750938812416}]
I20260809 06:46:35.877753 5195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:35.877997 5195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:35.883692 5195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70dee95e74b741408369a89d11608f0b: Sending DeleteTablet for 3 replicas of tablet 96984501ccfd4cd2babdbfd2d3e89e9d
I20260809 06:46:35.884960 6544 tablet_service.cc:1514] Processing DeleteTablet for tablet 96984501ccfd4cd2babdbfd2d3e89e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:35 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260809 06:46:35.885136 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 96984501ccfd4cd2babdbfd2d3e89e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:35 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:39358
I20260809 06:46:35.886435 6773 tablet_service.cc:1514] Processing DeleteTablet for tablet 96984501ccfd4cd2babdbfd2d3e89e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:35 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:55086
I20260809 06:46:35.886657 7194 tablet_replica.cc:330] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: stopping tablet replica
I20260809 06:46:35.886899 7187 tablet_replica.cc:330] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: stopping tablet replica
I20260809 06:46:35.887190 7194 raft_consensus.cc:2229] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:35.887321 7187 raft_consensus.cc:2229] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:35.887858 7194 raft_consensus.cc:2258] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:35.888254 7187 raft_consensus.cc:2258] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:35.888746 7195 tablet_replica.cc:330] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: stopping tablet replica
I20260809 06:46:35.889194 7195 raft_consensus.cc:2229] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:35.889752 7195 raft_consensus.cc:2258] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:35.892282 7194 ts_tablet_manager.cc:1905] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:35.892287 7187 ts_tablet_manager.cc:1905] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:35.897084 7195 ts_tablet_manager.cc:1905] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:35.899765 7194 ts_tablet_manager.cc:1918] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:35.900017 7194 log.cc:1196] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/96984501ccfd4cd2babdbfd2d3e89e9d
I20260809 06:46:35.900086 7187 ts_tablet_manager.cc:1918] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:35.900256 7187 log.cc:1196] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/96984501ccfd4cd2babdbfd2d3e89e9d
I20260809 06:46:35.900838 7194 ts_tablet_manager.cc:1939] T 96984501ccfd4cd2babdbfd2d3e89e9d P bbdce250244643f4968b48eab45d191c: Deleting consensus metadata
I20260809 06:46:35.900854 7187 ts_tablet_manager.cc:1939] T 96984501ccfd4cd2babdbfd2d3e89e9d P 22ddf774e2214df7a6f8e0c3ebca0377: Deleting consensus metadata
I20260809 06:46:35.902813 5168 catalog_manager.cc:4841] TS bbdce250244643f4968b48eab45d191c (127.27.209.195:43853): tablet 96984501ccfd4cd2babdbfd2d3e89e9d (table test-table [id=236aae700a9646f2979de9b7fbd5cb66]) successfully deleted
I20260809 06:46:35.904968 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 5147
I20260809 06:46:35.906152 5238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:35.906558 7195 ts_tablet_manager.cc:1918] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:35.906744 7195 log.cc:1196] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/96984501ccfd4cd2babdbfd2d3e89e9d
I20260809 06:46:35.907351 7195 ts_tablet_manager.cc:1939] T 96984501ccfd4cd2babdbfd2d3e89e9d P 228c3285dfbd49eaa4a9978cdc0464db: Deleting consensus metadata
I20260809 06:46:35.919855 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.890288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.27.209.253:33869)
0809 06:46:35.890460 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.890466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.901964 (+ 11498us) server_negotiation.cc:407] Connection header received
0809 06:46:35.902359 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.902366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.902458 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.902532 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.905160 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.906341 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.908227 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.908874 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.908973 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.913020 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.913028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.913033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.913074 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.916656 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.917475 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.917481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.917485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.917628 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.918009 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.918014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.918018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.918372 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.918594 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.919166 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.919416 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:46:35.937400 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:35.907943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.27.209.252:36221)
0809 06:46:35.908253 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:35.908261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:35.922124 (+ 13863us) server_negotiation.cc:407] Connection header received
0809 06:46:35.922376 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:35.922383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:35.922460 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:35.922521 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:35.924315 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.925458 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.927285 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:35.927908 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:35.928001 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:35.931399 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:35.931405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:35.931410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:35.931447 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:35.934579 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.935156 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.935160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.935162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.935305 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:35.935910 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:35.935914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:35.935916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:35.936186 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:35.936369 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:35.936859 (+ 490us) server_negotiation.cc:299] Negotiation successful
0809 06:46:35.937064 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260809 06:46:36.059080 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 5147
W20260809 06:46:36.084282 6469 connection.cc:535] server connection from 127.0.0.1:41704 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 06:46:36.125208 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:36.089063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.27.209.252:36221)
0809 06:46:36.089307 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:36.089315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:36.108646 (+ 19331us) server_negotiation.cc:407] Connection header received
0809 06:46:36.108990 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:36.108995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:36.109078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:36.109155 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:36.111248 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:36.112352 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:36.114252 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:36.114897 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:36.114993 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:36.118641 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:36.118648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:36.118652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:36.118698 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:36.122301 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:36.122943 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:36.122948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:36.122952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:36.123080 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:36.123657 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:36.123663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:36.123666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:36.123948 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:36.124189 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:36.124532 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 06:46:36.124773 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:46:36.128767 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:36.089557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.27.209.253:33869)
0809 06:46:36.089781 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:36.089788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:36.108354 (+ 18566us) server_negotiation.cc:407] Connection header received
0809 06:46:36.108945 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:36.108953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:36.109051 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:36.109130 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:36.116029 (+ 6899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:36.117117 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:36.119015 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:36.119726 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:36.119822 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:36.122655 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:36.122662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:36.122667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:36.122708 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:36.126132 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:36.126969 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:36.126974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:36.126976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:36.127133 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:36.127524 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:36.127527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:36.127529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:36.127841 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:36.128020 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:36.128377 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 06:46:36.128539 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 06:46:36.861958 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:36.861275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:45025)
0809 06:46:36.861684 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:36.861804 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":201,"threads_started":1}
W20260809 06:46:36.862376 6874 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:35661 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
I20260809 06:46:36.871264 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:36.870515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:44545)
0809 06:46:36.870992 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:36.871111 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260809 06:46:36.871384 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:36.870509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.195:46853)
0809 06:46:36.871054 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:36.871183 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":333,"threads_started":1}
W20260809 06:46:36.871677 6644 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:35661 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
W20260809 06:46:36.871804 7099 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:35661 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
I20260809 06:46:37.384299 7214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 70dee95e74b741408369a89d11608f0b)
I20260809 06:46:37.384529 7214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } }
I20260809 06:46:37.384939 7215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 70dee95e74b741408369a89d11608f0b)
I20260809 06:46:37.385211 7215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } }
I20260809 06:46:37.385571 7214 leader_election.cc:288] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661), 8094d282293f4843b5d01fb26cb0290b (127.27.209.253:33869)
I20260809 06:46:37.386979 7215 leader_election.cc:288] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661), 85f7189c52a64f1bb9c06f2bce3196f5 (127.27.209.252:36221)
I20260809 06:46:37.388073 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.385709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53166)
0809 06:46:37.387580 (+ 1871us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.387789 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1722,"threads_started":1}
I20260809 06:46:37.388504 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.387060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53182)
0809 06:46:37.387896 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.388118 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":582,"threads_started":1}
W20260809 06:46:37.388827 6062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
W20260809 06:46:37.395754 6337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
I20260809 06:46:37.398834 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.397942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53196)
0809 06:46:37.398377 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.398559 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307}
W20260809 06:46:37.399067 6062 leader_election.cc:334] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661): Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
I20260809 06:46:37.402037 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.401425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53204)
0809 06:46:37.401579 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.401789 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 06:46:37.402356 6337 leader_election.cc:334] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661): Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
I20260809 06:46:37.407111 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.385797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:33869 (local address 127.0.0.1:45018)
0809 06:46:37.388663 (+ 2866us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.388682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:37.388834 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:37.399872 (+ 11038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:37.399878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:37.399910 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:37.400211 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.400236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.402564 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.402589 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.404728 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.404746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.406307 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.406315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.406540 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.406578 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:46:37.406680 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 06:46:37.406825 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2775,"thread_start_us":1489,"threads_started":1}
I20260809 06:46:37.407330 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.385935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.27.209.253:33869)
0809 06:46:37.399432 (+ 13497us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.399438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.399462 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:46:37.399562 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.399567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.399657 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.399765 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:37.400412 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.401673 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.404949 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.406110 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.406269 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.406353 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.406776 (+ 423us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.406935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13379,"thread_start_us":13570,"threads_started":1}
I20260809 06:46:37.409380 6095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8094d282293f4843b5d01fb26cb0290b" is_pre_election: true
I20260809 06:46:37.409698 6095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f7189c52a64f1bb9c06f2bce3196f5 in term 1.
I20260809 06:46:37.410254 6338 leader_election.cc:302] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8094d282293f4843b5d01fb26cb0290b, 85f7189c52a64f1bb9c06f2bce3196f5; no voters: 70dee95e74b741408369a89d11608f0b
I20260809 06:46:37.410729 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.389389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:36221 (local address 127.0.0.1:36648)
0809 06:46:37.394099 (+ 4710us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.394121 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:37.394276 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:37.403927 (+ 9651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:37.403931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:37.403959 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:37.404274 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.404286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.406236 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.406241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.408328 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.408344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.410012 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.410019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.410203 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.410236 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:46:37.410324 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 06:46:37.410468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4599,"thread_start_us":8715,"threads_started":1}
I20260809 06:46:37.410739 7214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 06:46:37.410799 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.399921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.27.209.252:36221)
0809 06:46:37.401645 (+ 1724us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.401651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.401705 (+ 54us) server_negotiation.cc:407] Connection header received
0809 06:46:37.401823 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.401828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.401963 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.402102 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:37.404469 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.406058 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.408485 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.409670 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.409821 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.409886 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.410378 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.410511 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1514,"thread_start_us":1249,"threads_started":1}
I20260809 06:46:37.410856 7214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 70dee95e74b741408369a89d11608f0b)
I20260809 06:46:37.410974 7214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:46:37.411355 6374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8094d282293f4843b5d01fb26cb0290b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" is_pre_election: true
I20260809 06:46:37.413196 7214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } }
I20260809 06:46:37.413650 6374 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8094d282293f4843b5d01fb26cb0290b in current term 2: Already voted for candidate 85f7189c52a64f1bb9c06f2bce3196f5 in this term.
I20260809 06:46:37.413813 7214 leader_election.cc:288] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 2 election: Requested vote from peers 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661), 8094d282293f4843b5d01fb26cb0290b (127.27.209.253:33869)
I20260809 06:46:37.414391 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.413972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53224)
0809 06:46:37.414137 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.414225 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260809 06:46:37.414463 6095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8094d282293f4843b5d01fb26cb0290b"
I20260809 06:46:37.414726 6095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:46:37.415220 6065 leader_election.cc:302] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8094d282293f4843b5d01fb26cb0290b; no voters: 70dee95e74b741408369a89d11608f0b, 85f7189c52a64f1bb9c06f2bce3196f5
I20260809 06:46:37.416481 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.416126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53228)
0809 06:46:37.416288 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.416371 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 06:46:37.416649 6337 leader_election.cc:334] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661): Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
I20260809 06:46:37.417251 6095 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f7189c52a64f1bb9c06f2bce3196f5 in term 2.
I20260809 06:46:37.417470 7215 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:46:37.417662 6338 leader_election.cc:302] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8094d282293f4843b5d01fb26cb0290b, 85f7189c52a64f1bb9c06f2bce3196f5; no voters: 70dee95e74b741408369a89d11608f0b
I20260809 06:46:37.417956 7214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:46:37.418444 7214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [term 2 LEADER]: Becoming Leader. State: Replica: 85f7189c52a64f1bb9c06f2bce3196f5, State: Running, Role: LEADER
I20260809 06:46:37.418853 7214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } }
I20260809 06:46:37.420861 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f7189c52a64f1bb9c06f2bce3196f5. Latest consensus state: current_term: 2 leader_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:37.421228 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:37.421859 7233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:37.425586 7233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:37.426440 7233 catalog_manager.cc:1259] Loaded cluster ID: dd41263487264d9f942755b4bead47d3
I20260809 06:46:37.426523 7233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:37.427711 7233 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:37.428503 7233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Loaded TSK: 0
I20260809 06:46:37.429338 7233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:46:37.464124 6357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:37.471284 6357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:37.475674 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.475187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53246)
0809 06:46:37.475353 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.475439 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 06:46:37.475871 6095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [term 2 FOLLOWER]: Refusing update from remote peer 85f7189c52a64f1bb9c06f2bce3196f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 06:46:37.476619 7230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 06:46:37.476830 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.476468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53256)
0809 06:46:37.476642 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.476711 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 06:46:37.477033 6337 consensus_peers.cc:487] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 -> Peer 70dee95e74b741408369a89d11608f0b (127.27.209.254:35661): Couldn't send request to peer 70dee95e74b741408369a89d11608f0b. Status: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 06:46:37.479295 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f7189c52a64f1bb9c06f2bce3196f5. Latest consensus state: current_term: 2 leader_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:37.479588 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:46:37.481102 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:37.481413 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:37.481806 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "70dee95e74b741408369a89d11608f0b" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35661 } } peers { permanent_uuid: "8094d282293f4843b5d01fb26cb0290b" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 33869 } attrs { promote: false } } peers { permanent_uuid: "85f7189c52a64f1bb9c06f2bce3196f5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 36221 } attrs { promote: false } } }
I20260809 06:46:37.482085 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8094d282293f4843b5d01fb26cb0290b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:46:37.498790 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.490238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:34399 (local address 127.0.0.1:41720)
0809 06:46:37.490883 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.490903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:37.491028 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:37.491475 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:37.491479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:37.491503 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:37.491886 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.491900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.493864 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.493871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.496060 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.496078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.497916 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.497923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.498131 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.498176 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 06:46:37.498249 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 06:46:37.498411 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531}
I20260809 06:46:37.500053 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.490380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.27.209.193:34399)
0809 06:46:37.490880 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.490887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.491154 (+ 267us) server_negotiation.cc:407] Connection header received
0809 06:46:37.491235 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.491240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.491332 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.492041 (+ 709us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:37.492546 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.493683 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.496324 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.497642 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.499407 (+ 1765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.499495 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.499569 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.499731 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
I20260809 06:46:37.508896 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.491140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:45695 (local address 127.0.0.1:55100)
0809 06:46:37.499038 (+ 7898us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.499056 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:37.499174 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:37.503237 (+ 4063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:37.503242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:37.503269 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:37.503431 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.503441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.505000 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.505006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.506603 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.506627 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.507965 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.507972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.508229 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.508267 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:46:37.508373 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:46:37.508512 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7794,"thread_start_us":17003,"threads_started":1}
I20260809 06:46:37.509332 6544 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 (DEFAULT_TABLE table=test-table [id=d9c38962450a416e934248e79fc5297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:37.509786 6544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1e13d5babd4ad897e8878d0ba61f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:37.510030 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.491623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.27.209.194:45695)
0809 06:46:37.502802 (+ 11179us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.502808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.502828 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:46:37.502901 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.502906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.502985 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.503604 (+ 619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:37.503778 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.504832 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.506796 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.507763 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.507923 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.508002 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.509567 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.509741 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11048,"thread_start_us":11447,"threads_started":1}
I20260809 06:46:37.511274 6773 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 (DEFAULT_TABLE table=test-table [id=d9c38962450a416e934248e79fc5297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:37.511770 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1e13d5babd4ad897e8878d0ba61f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:37.512705 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.490732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:43853 (local address 127.0.0.1:39366)
0809 06:46:37.491121 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.491138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:37.491218 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:37.507288 (+ 16070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:37.507294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:37.507323 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:37.507494 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.507504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.508785 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.508789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.510513 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:37.510529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.511984 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.511990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:37.512167 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.512202 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:46:37.512294 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 06:46:37.512426 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260809 06:46:37.512787 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.491581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.27.209.195:43853)
0809 06:46:37.506898 (+ 15317us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.506905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.506931 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:46:37.507007 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.507013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.507107 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.507246 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:37.507710 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.508652 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.510737 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.511833 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.512001 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.512071 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.512343 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.512484 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15165,"thread_start_us":15792,"threads_started":1}
I20260809 06:46:37.513527 6998 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 (DEFAULT_TABLE table=test-table [id=d9c38962450a416e934248e79fc5297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:37.513969 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f1e13d5babd4ad897e8878d0ba61f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:37.515878 7252 tablet_bootstrap.cc:492] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Bootstrap starting.
I20260809 06:46:37.517457 7252 tablet_bootstrap.cc:654] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:37.519965 7254 tablet_bootstrap.cc:492] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Bootstrap starting.
I20260809 06:46:37.520408 7252 tablet_bootstrap.cc:492] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: No bootstrap required, opened a new log
I20260809 06:46:37.520673 7252 ts_tablet_manager.cc:1397] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 06:46:37.521497 7252 raft_consensus.cc:348] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.521754 7252 raft_consensus.cc:374] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:37.521806 7254 tablet_bootstrap.cc:654] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:37.521857 7252 raft_consensus.cc:729] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ddf774e2214df7a6f8e0c3ebca0377, State: Initialized, Role: FOLLOWER
I20260809 06:46:37.522094 7252 consensus_queue.cc:260] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.522948 7252 ts_tablet_manager.cc:1428] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:46:37.529444 7258 tablet_bootstrap.cc:492] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Bootstrap starting.
I20260809 06:46:37.530272 6646 heartbeater.cc:503] Master 127.27.209.252:36221 was elected leader, sending a full tablet report...
I20260809 06:46:37.533953 7258 tablet_bootstrap.cc:654] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:37.534545 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.523292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:39051)
0809 06:46:37.534150 (+ 10858us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.534303 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10722,"threads_started":1}
I20260809 06:46:37.535755 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.535320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:57735)
0809 06:46:37.535487 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.535616 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 06:46:37.536412 7254 tablet_bootstrap.cc:492] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: No bootstrap required, opened a new log
I20260809 06:46:37.536566 7258 tablet_bootstrap.cc:492] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: No bootstrap required, opened a new log
I20260809 06:46:37.536557 7254 ts_tablet_manager.cc:1397] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20260809 06:46:37.536711 7258 ts_tablet_manager.cc:1397] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260809 06:46:37.537423 7258 raft_consensus.cc:348] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.537372 7254 raft_consensus.cc:348] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.537606 7258 raft_consensus.cc:374] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:37.537614 7254 raft_consensus.cc:374] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:37.537706 7258 raft_consensus.cc:729] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdce250244643f4968b48eab45d191c, State: Initialized, Role: FOLLOWER
I20260809 06:46:37.537710 7254 raft_consensus.cc:729] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228c3285dfbd49eaa4a9978cdc0464db, State: Initialized, Role: FOLLOWER
I20260809 06:46:37.537963 7254 consensus_queue.cc:260] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.537976 7258 consensus_queue.cc:260] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.539067 7254 ts_tablet_manager.cc:1428] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260809 06:46:37.543830 7258 ts_tablet_manager.cc:1428] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260809 06:46:37.544080 7101 heartbeater.cc:503] Master 127.27.209.252:36221 was elected leader, sending a full tablet report...
I20260809 06:46:37.545220 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.539011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.195:36699)
0809 06:46:37.544882 (+ 5871us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.545002 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5724,"thread_start_us":5598,"threads_started":1}
I20260809 06:46:37.545326 6875 heartbeater.cc:503] Master 127.27.209.252:36221 was elected leader, sending a full tablet report...
I20260809 06:46:37.547819 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.547386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.195:54601)
0809 06:46:37.547581 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.547677 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 06:46:37.551414 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.539303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:47777)
0809 06:46:37.551080 (+ 11777us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.551221 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11675,"threads_started":1}
I20260809 06:46:37.552523 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.552167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:47385)
0809 06:46:37.552322 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.552413 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 06:46:37.580840 7259 raft_consensus.cc:482] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:37.581054 7259 raft_consensus.cc:504] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.581813 7259 leader_election.cc:288] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbdce250244643f4968b48eab45d191c (127.27.209.195:43853), 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:37.582365 7030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1e13d5babd4ad897e8878d0ba61f84" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdce250244643f4968b48eab45d191c" is_pre_election: true
I20260809 06:46:37.582393 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1e13d5babd4ad897e8878d0ba61f84" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228c3285dfbd49eaa4a9978cdc0464db" is_pre_election: true
I20260809 06:46:37.582599 6803 raft_consensus.cc:2454] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 0.
I20260809 06:46:37.582588 7030 raft_consensus.cc:2454] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 0.
I20260809 06:46:37.583082 6469 leader_election.cc:302] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ddf774e2214df7a6f8e0c3ebca0377, bbdce250244643f4968b48eab45d191c; no voters:
I20260809 06:46:37.583364 7259 raft_consensus.cc:2781] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:37.583472 7259 raft_consensus.cc:482] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:37.583583 7259 raft_consensus.cc:3037] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:37.585839 7259 raft_consensus.cc:504] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.586371 7259 leader_election.cc:288] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 election: Requested vote from peers bbdce250244643f4968b48eab45d191c (127.27.209.195:43853), 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695)
I20260809 06:46:37.586844 7030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1e13d5babd4ad897e8878d0ba61f84" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdce250244643f4968b48eab45d191c"
I20260809 06:46:37.586838 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1e13d5babd4ad897e8878d0ba61f84" candidate_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228c3285dfbd49eaa4a9978cdc0464db"
I20260809 06:46:37.587014 6803 raft_consensus.cc:3037] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:37.587013 7030 raft_consensus.cc:3037] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:37.589195 7030 raft_consensus.cc:2454] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 1.
I20260809 06:46:37.589185 6803 raft_consensus.cc:2454] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ddf774e2214df7a6f8e0c3ebca0377 in term 1.
I20260809 06:46:37.589526 6469 leader_election.cc:302] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ddf774e2214df7a6f8e0c3ebca0377, bbdce250244643f4968b48eab45d191c; no voters:
I20260809 06:46:37.589823 7259 raft_consensus.cc:2781] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:37.590230 7259 raft_consensus.cc:686] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 LEADER]: Becoming Leader. State: Replica: 22ddf774e2214df7a6f8e0c3ebca0377, State: Running, Role: LEADER
I20260809 06:46:37.590528 7259 consensus_queue.cc:237] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } }
I20260809 06:46:37.591184 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.590784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:54991)
0809 06:46:37.590933 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.591026 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 06:46:37.593114 6357 catalog_manager.cc:5495] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ddf774e2214df7a6f8e0c3ebca0377" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 34399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 } health_report { overall_health: UNKNOWN } } }
I20260809 06:46:37.615211 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.611091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.27.209.193:34399)
0809 06:46:37.611250 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.611255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.611273 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:46:37.611400 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.611403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.611467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.611564 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:37.611858 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.612543 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.613832 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.614258 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.614350 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.614654 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:37.614763 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:37.614829 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.614926 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.614986 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 06:46:37.618180 6803 raft_consensus.cc:1261] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Refusing update from remote peer 22ddf774e2214df7a6f8e0c3ebca0377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:37.618216 7030 raft_consensus.cc:1261] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Refusing update from remote peer 22ddf774e2214df7a6f8e0c3ebca0377: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:37.618846 7268 consensus_queue.cc:1035] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "228c3285dfbd49eaa4a9978cdc0464db" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:37.619146 7259 consensus_queue.cc:1035] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbdce250244643f4968b48eab45d191c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:37.621639 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.621180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:46389)
0809 06:46:37.621324 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.621427 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 06:46:37.622051 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.621563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.195:52015)
0809 06:46:37.621728 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.621858 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 06:46:37.627281 7277 mvcc.cc:205] Tried to move back new op lower bound from 7316512758238568448 to 7316512758132944896. Current Snapshot: MvccSnapshot[applied={T|T < 7316512758238568448}]
I20260809 06:46:37.629664 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.629207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:41969)
0809 06:46:37.629381 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.629485 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 06:46:37.633625 6358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:37.633956 6358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:37.637632 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.637175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.193:32869)
0809 06:46:37.637378 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.637467 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260809 06:46:37.638492 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.638027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.27.209.194:41365)
0809 06:46:37.638249 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.638341 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260809 06:46:37.641134 6358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f7189c52a64f1bb9c06f2bce3196f5: Sending DeleteTablet for 3 replicas of tablet 1f1e13d5babd4ad897e8878d0ba61f84
I20260809 06:46:37.642098 6544 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:37 UTC) from {username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:41720
I20260809 06:46:37.642181 6998 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:37 UTC) from {username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:39366
I20260809 06:46:37.642334 6773 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:37 UTC) from {username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:55100
I20260809 06:46:37.642474 6772 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1e13d5babd4ad897e8878d0ba61f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:37 UTC) from {username='oryx', principal='oryx/127.27.209.252@KRBTEST.COM'} at 127.0.0.1:55100
I20260809 06:46:37.643978 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 6325
I20260809 06:46:37.651705 7288 tablet_replica.cc:330] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: stopping tablet replica
I20260809 06:46:37.652673 7288 raft_consensus.cc:2229] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:37.653188 7288 raft_consensus.cc:2258] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:37.654491 7288 ts_tablet_manager.cc:1905] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:37.654832 7286 tablet_replica.cc:330] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: stopping tablet replica
I20260809 06:46:37.655010 7286 raft_consensus.cc:2229] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:37.655362 7286 raft_consensus.cc:2258] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:37.655843 7287 tablet_replica.cc:330] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: stopping tablet replica
I20260809 06:46:37.656021 7287 raft_consensus.cc:2229] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:37.656276 7287 raft_consensus.cc:2258] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:37.657262 7287 ts_tablet_manager.cc:1905] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:37.657902 7286 ts_tablet_manager.cc:1905] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:37.660315 7288 ts_tablet_manager.cc:1918] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:37.660516 7288 log.cc:1196] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/1f1e13d5babd4ad897e8878d0ba61f84
I20260809 06:46:37.661033 7288 ts_tablet_manager.cc:1939] T 1f1e13d5babd4ad897e8878d0ba61f84 P 228c3285dfbd49eaa4a9978cdc0464db: Deleting consensus metadata
I20260809 06:46:37.662343 6337 catalog_manager.cc:4841] TS 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695): tablet 1f1e13d5babd4ad897e8878d0ba61f84 (table test-table [id=d9c38962450a416e934248e79fc5297d]) successfully deleted
W20260809 06:46:37.663497 6337 catalog_manager.cc:4808] TS 228c3285dfbd49eaa4a9978cdc0464db (127.27.209.194:45695): delete failed for tablet 1f1e13d5babd4ad897e8878d0ba61f84 because the tablet was not found. No further retry: Not found: Tablet not found: 1f1e13d5babd4ad897e8878d0ba61f84
I20260809 06:46:37.663805 7287 ts_tablet_manager.cc:1918] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:37.663931 7287 log.cc:1196] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/1f1e13d5babd4ad897e8878d0ba61f84
I20260809 06:46:37.664307 7286 ts_tablet_manager.cc:1918] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:37.664350 7287 ts_tablet_manager.cc:1939] T 1f1e13d5babd4ad897e8878d0ba61f84 P bbdce250244643f4968b48eab45d191c: Deleting consensus metadata
I20260809 06:46:37.664449 7286 log.cc:1196] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/1f1e13d5babd4ad897e8878d0ba61f84
I20260809 06:46:37.664865 7286 ts_tablet_manager.cc:1939] T 1f1e13d5babd4ad897e8878d0ba61f84 P 22ddf774e2214df7a6f8e0c3ebca0377: Deleting consensus metadata
I20260809 06:46:37.665505 6339 catalog_manager.cc:4841] TS bbdce250244643f4968b48eab45d191c (127.27.209.195:43853): tablet 1f1e13d5babd4ad897e8878d0ba61f84 (table test-table [id=d9c38962450a416e934248e79fc5297d]) successfully deleted
I20260809 06:46:37.665877 6337 catalog_manager.cc:4841] TS 22ddf774e2214df7a6f8e0c3ebca0377 (127.27.209.193:34399): tablet 1f1e13d5babd4ad897e8878d0ba61f84 (table test-table [id=d9c38962450a416e934248e79fc5297d]) successfully deleted
I20260809 06:46:37.670502 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.644956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.27.209.193:34399)
0809 06:46:37.645110 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.645118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.652005 (+ 6887us) server_negotiation.cc:407] Connection header received
0809 06:46:37.652194 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.652211 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.652309 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.652436 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:37.657079 (+ 4643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.658113 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.661261 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.661894 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.662006 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.665477 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:37.665496 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:37.665515 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:37.665562 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:37.668228 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:37.668855 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:37.668863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:37.668865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:37.668998 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:37.669280 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:37.669283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:37.669285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:37.669634 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:37.669851 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.670082 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.670240 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 06:46:37.671084 6637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:37.788997 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 6325
I20260809 06:46:37.815217 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 6666
I20260809 06:46:37.827960 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.816059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.27.209.194:45695)
0809 06:46:37.816235 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.816239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.816258 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:46:37.816408 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.816412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.816476 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.816569 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:37.817778 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.818459 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.819725 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.820181 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.820285 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.822730 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:37.822746 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:37.822765 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:37.822797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:37.825660 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:37.826295 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:37.826303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:37.826305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:37.826432 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:37.826719 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:37.826722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:37.826724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:37.827038 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:37.827279 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.827454 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.827694 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:46:37.828660 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:37.899514 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.899009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53266)
0809 06:46:37.899193 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.899345 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 06:46:37.900554 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.900226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35661 (local address 127.0.0.1:53268)
0809 06:46:37.900346 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:37.900423 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:35661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 06:46:37.920840 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 6666
I20260809 06:46:37.943073 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 6901
I20260809 06:46:37.955868 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:37.943888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.27.209.195:43853)
0809 06:46:37.944049 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:37.944054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:37.944117 (+ 63us) server_negotiation.cc:407] Connection header received
0809 06:46:37.944294 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:37.944298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:37.944366 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:37.944452 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:37.945749 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.946433 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.947718 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:37.948121 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:37.948213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:37.950595 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:37.950613 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:37.950631 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:37.950662 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:37.953471 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:37.954047 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:37.954054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:37.954057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:37.954182 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:37.954497 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:37.954500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:37.954502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:37.954908 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:37.955135 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:37.955367 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 06:46:37.955580 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:46:37.956468 7093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:38.048418 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 6901
I20260809 06:46:38.070623 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 5882
I20260809 06:46:38.071815 6120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:38.187898 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 5882
I20260809 06:46:38.214344 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 6035
I20260809 06:46:38.215438 6408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:38.336052 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 6035
2026-08-09T06:46:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9533 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:38 dist-test-slave-jpj1 krb5kdc[7303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:38 dist-test-slave-jpj1 krb5kdc[7303](info): set up 2 sockets
Aug 09 06:46:38 dist-test-slave-jpj1 krb5kdc[7303](info): commencing operation
krb5kdc: starting...
W20260809 06:46:39.537825 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:39 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab.
Entry for principal oryx/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab.
I20260809 06:46:39.772333 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40897
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab
--principal=oryx/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:40.095619 7321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:40.095920 7321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:40.096040 7321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:40.103109 7321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:40.103205 7321 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 06:46:40.103266 7321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:40.103353 7321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:40.103420 7321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:40.103505 7321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:40.113095 7321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:40897
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.254.keytab
--principal=oryx/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:40.114323 7321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:40.115643 7321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:40.126609 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:40.126732 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:40.127094 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:40.127830 7321 server_base.cc:1034] running on GCE node
I20260809 06:46:40.128583 7321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:40.129906 7321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:40.131124 7321 hybrid_clock.cc:648] HybridClock initialized: now 1786258000131066 us; error 28 us; skew 500 ppm
I20260809 06:46:40.131436 7321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:40 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258000, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:40.134691 7321 init.cc:377] Logged in from keytab as oryx/127.27.209.254@KRBTEST.COM (short username oryx)
I20260809 06:46:40.136230 7321 webserver.cc:460] Webserver started at http://127.27.209.254:46057/ using document root <none> and password file <none>
I20260809 06:46:40.136919 7321 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:40.137015 7321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:40.137338 7321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:40.140327 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "42990dade52942dd9db38a0b7c48c0cf"
format_stamp: "Formatted at 2026-08-09 06:46:40 on dist-test-slave-jpj1"
server_key: "79ab0c0daf7c780ac2fecbccc4f1f375"
server_key_iv: "3bc206c967352d492647eb3fde876a0d"
server_key_version: "encryptionkey@0"
I20260809 06:46:40.141098 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "42990dade52942dd9db38a0b7c48c0cf"
format_stamp: "Formatted at 2026-08-09 06:46:40 on dist-test-slave-jpj1"
server_key: "79ab0c0daf7c780ac2fecbccc4f1f375"
server_key_iv: "3bc206c967352d492647eb3fde876a0d"
server_key_version: "encryptionkey@0"
I20260809 06:46:40.145982 7321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 06:46:40.149317 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:40.150538 7321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:46:40.150735 7321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "42990dade52942dd9db38a0b7c48c0cf"
format_stamp: "Formatted at 2026-08-09 06:46:40 on dist-test-slave-jpj1"
server_key: "79ab0c0daf7c780ac2fecbccc4f1f375"
server_key_iv: "3bc206c967352d492647eb3fde876a0d"
server_key_version: "encryptionkey@0"
I20260809 06:46:40.150926 7321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:40.169880 7321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:40.173851 7321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:40.174105 7321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:40.193231 7321 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:40897
I20260809 06:46:40.193251 7436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:40897 every 8 connection(s)
I20260809 06:46:40.194584 7321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:40.198484 7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:40.203344 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 7321
I20260809 06:46:40.203524 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:40.204205 28487 external_mini_cluster.cc:1428] Setting key 5381262785565220e8d4e1e6eedbd95f
I20260809 06:46:40.210633 7437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Bootstrap starting.
I20260809 06:46:40.214437 7437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:40.215617 7437 log.cc:824] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:40.218503 7437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: No bootstrap required, opened a new log
Aug 09 06:46:40 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786257999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:40.228628 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:40.206126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.27.209.254:40897)
0809 06:46:40.206965 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:40.206980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:40.207036 (+ 56us) server_negotiation.cc:407] Connection header received
0809 06:46:40.209492 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:40.209522 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:40.210045 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:40.210494 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:40.211806 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.213816 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.215124 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.216205 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.216391 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:40.220264 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:40.220275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:40.220297 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:40.220350 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:40.224291 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:40.224935 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:40.224945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:40.224956 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:40.225176 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:40.225487 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:40.225492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:40.225495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:40.226207 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:40.226518 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:40.226810 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 06:46:40.227263 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":307,"threads_started":1}
I20260809 06:46:40.228595 7437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42990dade52942dd9db38a0b7c48c0cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40897 } }
I20260809 06:46:40.229035 7437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:40.229125 7437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42990dade52942dd9db38a0b7c48c0cf, State: Initialized, Role: FOLLOWER
I20260809 06:46:40.229759 7437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42990dade52942dd9db38a0b7c48c0cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40897 } }
I20260809 06:46:40.230006 7437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:40.230167 7437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:40.230351 7437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:40.233027 7437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42990dade52942dd9db38a0b7c48c0cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40897 } }
I20260809 06:46:40.233467 7437 leader_election.cc:302] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42990dade52942dd9db38a0b7c48c0cf; no voters:
I20260809 06:46:40.233961 7437 leader_election.cc:288] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:40.234088 7445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:40.234728 7445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [term 1 LEADER]: Becoming Leader. State: Replica: 42990dade52942dd9db38a0b7c48c0cf, State: Running, Role: LEADER
I20260809 06:46:40.235213 7445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42990dade52942dd9db38a0b7c48c0cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40897 } }
I20260809 06:46:40.235967 7437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:40.239830 7446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42990dade52942dd9db38a0b7c48c0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42990dade52942dd9db38a0b7c48c0cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40897 } } }
I20260809 06:46:40.240330 7446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:40.239854 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42990dade52942dd9db38a0b7c48c0cf. Latest consensus state: current_term: 1 leader_uuid: "42990dade52942dd9db38a0b7c48c0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42990dade52942dd9db38a0b7c48c0cf" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40897 } } }
I20260809 06:46:40.241120 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:40.243382 7463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:40.247867 7463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:40.256767 7463 catalog_manager.cc:1347] Generated new cluster ID: fe8d3d910288486d8ba2d73b78b501ff
I20260809 06:46:40.256873 7463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:40.287662 7463 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:40.289511 7463 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:40.303058 7463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Generated new TSK 0
I20260809 06:46:40.304065 7463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab.
Entry for principal oryx/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab.
I20260809 06:46:40.366720 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40897
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:40.690762 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:40.691035 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:40.691174 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:40.698233 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:40.698339 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:40.698468 7479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:46:40.698575 7479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:40.707996 7479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:40.709194 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:40.710480 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:40.722859 7494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:40.722898 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:40.723326 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:40.723747 7479 server_base.cc:1034] running on GCE node
I20260809 06:46:40.724325 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:40.725138 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:40.726361 7479 hybrid_clock.cc:648] HybridClock initialized: now 1786258000726291 us; error 42 us; skew 500 ppm
I20260809 06:46:40.726652 7479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:40 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258000, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:40.729674 7479 init.cc:377] Logged in from keytab as oryx/127.27.209.193@KRBTEST.COM (short username oryx)
I20260809 06:46:40.730930 7479 webserver.cc:460] Webserver started at http://127.27.209.193:44813/ using document root <none> and password file <none>
I20260809 06:46:40.731588 7479 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:40.731695 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:40.731969 7479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:40.734671 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "4783a9e155fd41b3a72ef50eef248262"
format_stamp: "Formatted at 2026-08-09 06:46:40 on dist-test-slave-jpj1"
server_key: "f71c57f9eee4434804c3571434fe255c"
server_key_iv: "4d707890475f8846da1be05675098fc3"
server_key_version: "encryptionkey@0"
I20260809 06:46:40.735390 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "4783a9e155fd41b3a72ef50eef248262"
format_stamp: "Formatted at 2026-08-09 06:46:40 on dist-test-slave-jpj1"
server_key: "f71c57f9eee4434804c3571434fe255c"
server_key_iv: "4d707890475f8846da1be05675098fc3"
server_key_version: "encryptionkey@0"
I20260809 06:46:40.740276 7479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 06:46:40.743709 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:40.744927 7479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:46:40.745076 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "4783a9e155fd41b3a72ef50eef248262"
format_stamp: "Formatted at 2026-08-09 06:46:40 on dist-test-slave-jpj1"
server_key: "f71c57f9eee4434804c3571434fe255c"
server_key_iv: "4d707890475f8846da1be05675098fc3"
server_key_version: "encryptionkey@0"
I20260809 06:46:40.745285 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:40.760454 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:40.764119 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:40.764361 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:40.766839 7479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:40.766923 7479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:40.767046 7479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:40.767115 7479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:46:40 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258000, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:40.787977 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:40.767355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:36931 (local address 127.27.209.254:40897)
0809 06:46:40.768016 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:40.768023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:40.768975 (+ 952us) server_negotiation.cc:407] Connection header received
0809 06:46:40.770967 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:40.770974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:40.771065 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:40.771189 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:40.773230 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.774171 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.776570 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.776986 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.777096 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:40.781674 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:40.781682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:40.781687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:40.781734 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:40.784875 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:40.785642 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:40.785647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:40.785651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:40.785804 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:40.786321 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:40.786327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:40.786329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:40.786596 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:40.786728 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:40.787371 (+ 643us) server_negotiation.cc:299] Negotiation successful
0809 06:46:40.787622 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":422,"threads_started":1}
I20260809 06:46:40.788956 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:40.767961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.27.209.193:36931)
0809 06:46:40.768800 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:40.768844 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:40.770692 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:40.771449 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:40.771457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:40.772178 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:40.773033 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:40.773062 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.774320 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.774324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:40.776432 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:40.776444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.777225 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.777233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:40.777570 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:40.778336 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:40.778373 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:40.781424 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:40.785002 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:40.785008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:40.785020 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:40.785502 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:40.785953 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:40.785958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:40.785970 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:40.786196 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:40.786718 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:40.786725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:40.787032 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 06:46:40.787252 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 06:46:40.787761 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":344,"threads_started":1}
I20260809 06:46:40.815732 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:40.798686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.27.209.193:49015)
0809 06:46:40.802672 (+ 3986us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:40.802692 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:40.802849 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:40.803211 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:40.803216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:40.803803 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:40.804209 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:40.804226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.805400 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.805404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:40.807164 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:40.807176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.808035 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.808041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:40.808261 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:40.808994 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:40.809018 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:40.810115 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:40.813076 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:40.813082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:40.813086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:40.813483 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:40.813857 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:40.813862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:40.813866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:40.814027 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:40.814694 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:40.814698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:40.814899 (+ 201us) client_negotiation.cc:769] Sending connection context
0809 06:46:40.815062 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 06:46:40.815338 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"thread_start_us":3642,"threads_started":1}
I20260809 06:46:40.815850 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:40.798573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:49015 (local address 127.27.209.254:40897)
0809 06:46:40.798708 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:40.798712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:40.802800 (+ 4088us) server_negotiation.cc:407] Connection header received
0809 06:46:40.802966 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:40.802972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:40.803060 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:40.803174 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:40.804421 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.805266 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.807296 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:40.807832 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:40.807960 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:40.810290 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:40.810295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:40.810298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:40.810328 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:40.812959 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:40.813586 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:40.813592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:40.813596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:40.813725 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:40.814167 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:40.814171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:40.814173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:40.814501 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:40.814641 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:40.815224 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 06:46:40.815471 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 06:46:40.823529 7479 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:40911
I20260809 06:46:40.823625 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:40911 every 8 connection(s)
I20260809 06:46:40.824740 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:40.826514 7691 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40897
I20260809 06:46:40.826717 7691 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:40.827332 7691 heartbeater.cc:511] Master 127.27.209.254:40897 requested a full tablet report, sending...
I20260809 06:46:40.829119 7373 ts_manager.cc:194] Registered new tserver with Master: 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911)
I20260809 06:46:40.830382 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 7479
I20260809 06:46:40.830552 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:40.830482 7373 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:36931
I20260809 06:46:40.831169 28487 external_mini_cluster.cc:1428] Setting key dd367dd3c4ce69622ee97d3e1ed40f76
WARNING: no policy specified for oryx/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab.
Entry for principal oryx/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab.
I20260809 06:46:40.879123 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40897
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:41.204582 7697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:41.204929 7697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:41.205076 7697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:41.216080 7697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:41.216194 7697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:41.216287 7697 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:46:41.216367 7697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:41.225971 7697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:41.227207 7697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:41.228539 7697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:41.241222 7703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:41.241240 7704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:41.241880 7706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:41.242997 7697 server_base.cc:1034] running on GCE node
I20260809 06:46:41.243664 7697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:41.244570 7697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:41.245807 7697 hybrid_clock.cc:648] HybridClock initialized: now 1786258001245733 us; error 48 us; skew 500 ppm
I20260809 06:46:41.246107 7697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258001, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:41.249277 7697 init.cc:377] Logged in from keytab as oryx/127.27.209.194@KRBTEST.COM (short username oryx)
I20260809 06:46:41.250733 7697 webserver.cc:460] Webserver started at http://127.27.209.194:41525/ using document root <none> and password file <none>
I20260809 06:46:41.251446 7697 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:41.251575 7697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:41.251904 7697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:41.254808 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "6aed8871ee87489784eb3153555979be"
format_stamp: "Formatted at 2026-08-09 06:46:41 on dist-test-slave-jpj1"
server_key: "58ea47a3aedb3a4b3a373fcb9b274bee"
server_key_iv: "8661ebb1022ed002ac458ad19795edfb"
server_key_version: "encryptionkey@0"
I20260809 06:46:41.255648 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "6aed8871ee87489784eb3153555979be"
format_stamp: "Formatted at 2026-08-09 06:46:41 on dist-test-slave-jpj1"
server_key: "58ea47a3aedb3a4b3a373fcb9b274bee"
server_key_iv: "8661ebb1022ed002ac458ad19795edfb"
server_key_version: "encryptionkey@0"
I20260809 06:46:41.260775 7697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:46:41.265066 7718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:41.266413 7697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 06:46:41.266566 7697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "6aed8871ee87489784eb3153555979be"
format_stamp: "Formatted at 2026-08-09 06:46:41 on dist-test-slave-jpj1"
server_key: "58ea47a3aedb3a4b3a373fcb9b274bee"
server_key_iv: "8661ebb1022ed002ac458ad19795edfb"
server_key_version: "encryptionkey@0"
I20260809 06:46:41.266780 7697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:41.294488 7697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:41.298239 7697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:41.298487 7697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:41.301170 7697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:41.301273 7697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:41.301461 7697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:41.301533 7697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258001, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:41.322454 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.301559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:49003 (local address 127.27.209.254:40897)
0809 06:46:41.301712 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.301717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.303129 (+ 1412us) server_negotiation.cc:407] Connection header received
0809 06:46:41.305154 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.305160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.305242 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.305329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:41.307308 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.308295 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.310799 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.311336 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.311446 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.315915 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:41.315922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:41.315925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:41.315981 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:41.318541 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.319458 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.319464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.319468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.319654 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.320104 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.320109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.320113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.320475 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:41.320622 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.321787 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.322043 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 06:46:41.322926 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.302236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.27.209.194:49003)
0809 06:46:41.302981 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.303016 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.304889 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.305571 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.305580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.306245 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:41.307067 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.307097 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.308487 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.308491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.310643 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.310658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.311579 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.311588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.311835 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.312555 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:41.312589 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:41.315708 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:41.318667 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.318679 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.318695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.319330 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.319782 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.319795 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.319798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.319992 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.320593 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:41.320601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:41.320954 (+ 353us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.321297 (+ 343us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.321772 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":333,"threads_started":1}
I20260809 06:46:41.342139 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.328262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:35927 (local address 127.27.209.254:40897)
0809 06:46:41.328489 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.328496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.329033 (+ 537us) server_negotiation.cc:407] Connection header received
0809 06:46:41.329192 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.329195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.329262 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.329342 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:41.330513 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.331199 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.333324 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.333854 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.333976 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.336411 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:41.336416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:41.336419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:41.336450 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:41.338943 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.339797 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.339803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.339807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.339965 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.340353 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.340357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.340361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.340797 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:41.340985 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.341575 (+ 590us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.341812 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:46:41.342404 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.328235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.27.209.194:35927)
0809 06:46:41.328921 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.328943 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.329090 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.329384 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.329389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.329988 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:41.330386 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.330398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.331309 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.331314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.333081 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.333099 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.334059 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.334069 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.334236 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.334876 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:41.334900 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:41.336221 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:41.339062 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.339068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.339072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.339656 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.340092 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.340096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.340102 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.340241 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.340993 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:41.341000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:41.341261 (+ 261us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.341435 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.341846 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":385,"threads_started":1}
I20260809 06:46:41.353287 7697 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:38947
I20260809 06:46:41.353349 7912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:38947 every 8 connection(s)
I20260809 06:46:41.354493 7697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:41.356508 7914 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40897
I20260809 06:46:41.356693 7914 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:41.357321 7914 heartbeater.cc:511] Master 127.27.209.254:40897 requested a full tablet report, sending...
I20260809 06:46:41.358553 7373 ts_manager.cc:194] Registered new tserver with Master: 6aed8871ee87489784eb3153555979be (127.27.209.194:38947)
I20260809 06:46:41.359162 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 7697
I20260809 06:46:41.359309 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:41.359388 7373 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:49003
I20260809 06:46:41.359879 28487 external_mini_cluster.cc:1428] Setting key 72c06d8984f11061101d15e1b10d61c4
WARNING: no policy specified for oryx/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab.
Entry for principal oryx/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab.
I20260809 06:46:41.408290 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:40897
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:41.730808 7923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:41.731062 7923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:41.731156 7923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:41.738619 7923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:41.738734 7923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:41.738816 7923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 06:46:41.738896 7923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:41.748292 7923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.209.254:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:41.749516 7923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:41.750941 7923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:41.763506 7935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:41.763579 7936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:41.763554 7939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:41.764400 7923 server_base.cc:1034] running on GCE node
I20260809 06:46:41.765072 7923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:41.765934 7923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:41.767148 7923 hybrid_clock.cc:648] HybridClock initialized: now 1786258001767074 us; error 47 us; skew 500 ppm
I20260809 06:46:41.767438 7923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258001, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:41.770459 7923 init.cc:377] Logged in from keytab as oryx/127.27.209.195@KRBTEST.COM (short username oryx)
I20260809 06:46:41.772606 7923 webserver.cc:460] Webserver started at http://127.27.209.195:41373/ using document root <none> and password file <none>
I20260809 06:46:41.773277 7923 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:41.773378 7923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:41.773686 7923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:41.776520 7923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
format_stamp: "Formatted at 2026-08-09 06:46:41 on dist-test-slave-jpj1"
server_key: "81f23d4c0b7bbe8ab8aef12f913008d7"
server_key_iv: "029f9bb52ca09701d05a1b847dd7329a"
server_key_version: "encryptionkey@0"
I20260809 06:46:41.777280 7923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
format_stamp: "Formatted at 2026-08-09 06:46:41 on dist-test-slave-jpj1"
server_key: "81f23d4c0b7bbe8ab8aef12f913008d7"
server_key_iv: "029f9bb52ca09701d05a1b847dd7329a"
server_key_version: "encryptionkey@0"
I20260809 06:46:41.782088 7923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 06:46:41.785305 7947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:41.786533 7923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 06:46:41.786674 7923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
format_stamp: "Formatted at 2026-08-09 06:46:41 on dist-test-slave-jpj1"
server_key: "81f23d4c0b7bbe8ab8aef12f913008d7"
server_key_iv: "029f9bb52ca09701d05a1b847dd7329a"
server_key_version: "encryptionkey@0"
I20260809 06:46:41.786860 7923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:41.812484 7923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:41.816293 7923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:41.816551 7923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:41.819077 7923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:41.819168 7923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:41.819303 7923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:41.819363 7923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258001, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:41.833477 7691 heartbeater.cc:503] Master 127.27.209.254:40897 was elected leader, sending a full tablet report...
I20260809 06:46:41.841408 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.819708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:37591 (local address 127.27.209.254:40897)
0809 06:46:41.819917 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.819924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.821301 (+ 1377us) server_negotiation.cc:407] Connection header received
0809 06:46:41.823305 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.823311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.823412 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.823490 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:41.826038 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.827139 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.828978 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.829591 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.829722 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.834368 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:41.834374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:41.834378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:41.834420 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:41.837614 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.838531 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.838535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.838538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.838690 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.839168 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.839173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.839177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.839461 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:41.839634 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.840646 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.840934 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 06:46:41.842897 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.820247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.27.209.195:37591)
0809 06:46:41.821132 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.821176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.823005 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.824056 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.824066 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.824840 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:41.825803 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.825833 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.827357 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.827366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.828835 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.828847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.829961 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.829967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.830151 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.830823 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:41.830855 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:41.834090 (+ 3235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:41.837739 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.837749 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.837765 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.838371 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.838790 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.838796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.838800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.839047 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.839588 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:41.839596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:41.840090 (+ 494us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.840442 (+ 352us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.841034 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":344,"threads_started":1}
I20260809 06:46:41.862177 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.848382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:53523 (local address 127.27.209.254:40897)
0809 06:46:41.848602 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.848609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.849146 (+ 537us) server_negotiation.cc:407] Connection header received
0809 06:46:41.849503 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.849509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.849656 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.849734 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:41.851132 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.852275 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.854126 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.854671 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.854802 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.856797 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:41.856801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:41.856804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:41.856841 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:41.859212 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.859979 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.859985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.859989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.860157 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.860596 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.860601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.860605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.860957 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:41.861110 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.861602 (+ 492us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.861834 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:46:41.862273 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.848409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.27.209.195:53523)
0809 06:46:41.849039 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.849062 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.849216 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.849954 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.849960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.850548 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:41.850969 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.850983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.852419 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.852425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.853905 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.853921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.854877 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.854886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.855148 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.855846 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:41.855893 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:41.856627 (+ 734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:41.859343 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.859349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.859353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.859813 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.860289 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:41.860293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:41.860295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:41.860450 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:41.861107 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:41.861116 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:41.861349 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.861478 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.861855 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":337,"threads_started":1}
I20260809 06:46:41.875939 7923 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:33691
I20260809 06:46:41.875980 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:33691 every 8 connection(s)
I20260809 06:46:41.877105 7923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:41.879050 8141 heartbeater.cc:347] Connected to a master server at 127.27.209.254:40897
I20260809 06:46:41.879251 8141 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:41.879914 8141 heartbeater.cc:511] Master 127.27.209.254:40897 requested a full tablet report, sending...
I20260809 06:46:41.881201 7373 ts_manager.cc:194] Registered new tserver with Master: 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:41.882035 7373 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:37591
I20260809 06:46:41.883021 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 7923
I20260809 06:46:41.883172 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:41.883625 28487 external_mini_cluster.cc:1428] Setting key abd81766215194a09284db05bb1a22fd
I20260809 06:46:41.886305 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:46:41.940160 7516 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:41.929919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.27.209.254:40897)
0809 06:46:41.930126 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.930133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.930393 (+ 260us) server_negotiation.cc:407] Connection header received
0809 06:46:41.930576 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.930580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.930649 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.930742 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:41.932129 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.932857 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.934112 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.934549 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.934660 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.939854 (+ 5194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37750: BlockingRecv error: recv got EOF from 127.0.0.1:37750 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 06:46:41 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:41.955384 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.943039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.27.209.254:40897)
0809 06:46:41.943178 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.943182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.943571 (+ 389us) server_negotiation.cc:407] Connection header received
0809 06:46:41.943720 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.943723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.943797 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.943871 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:41.944892 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.945664 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.946799 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.947194 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.947310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.950671 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:41.950676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:41.950681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:41.950714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:41.952917 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.953516 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.953520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.953522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.953641 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:41.954035 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:41.954040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:41.954044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:41.954311 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:41.954615 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.954859 (+ 244us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.955099 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 06:46:41.959146 7373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:41.965929 7373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:41.997705 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.977606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40911 (local address 127.0.0.1:42368)
0809 06:46:41.982305 (+ 4699us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.982330 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.982452 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.989225 (+ 6773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.989234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.989269 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:41.989773 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.989788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.992284 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.992289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.994769 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.994785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.996336 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.996346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.996887 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.996951 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.997095 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.997270 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":342,"thread_start_us":714,"threads_started":1}
I20260809 06:46:41.997711 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.978179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.27.209.193:40911)
0809 06:46:41.988284 (+ 10105us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.988295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.988329 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:46:41.988439 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.988447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.988769 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.990075 (+ 1306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:41.990324 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.992127 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.994950 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.996099 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.996559 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.996746 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.997256 (+ 510us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.997421 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9922,"thread_start_us":10745,"threads_started":1}
I20260809 06:46:41.997828 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.978425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33691 (local address 127.0.0.1:56780)
0809 06:46:41.988997 (+ 10572us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.989015 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.989138 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.991291 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.991296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.991319 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:41.991691 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.991706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.993707 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.993711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.995165 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.995184 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.997092 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.997098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.997262 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.997291 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.997372 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.997493 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10451,"mutex_wait_us":47,"thread_start_us":10732,"threads_started":1}
I20260809 06:46:41.998428 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.978958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.27.209.195:33691)
0809 06:46:41.990818 (+ 11860us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.990824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.990851 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:46:41.990923 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.990928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.991100 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.991289 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:41.992050 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.993575 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.995343 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.996771 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.996959 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.997135 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.997946 (+ 811us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.998102 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11680,"thread_start_us":11500,"threads_started":1}
I20260809 06:46:41.999413 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.977162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38947 (local address 127.0.0.1:41928)
0809 06:46:41.977828 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:41.977870 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:41.978024 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:41.992573 (+ 14549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:41.992578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:41.992607 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:41.993091 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.993104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.995013 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.995018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.997022 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:41.997044 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.998699 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.998706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:41.998856 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.998886 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:46:41.998949 (+ 63us) client_negotiation.cc:240] Negotiation successful
0809 06:46:41.999064 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":474,"threads_started":1}
I20260809 06:46:41.999989 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:41.977388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.27.209.194:38947)
0809 06:46:41.992042 (+ 14654us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:41.992048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:41.992069 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:46:41.992147 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:41.992152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:41.992358 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:41.993319 (+ 961us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:41.993502 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.994855 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.997217 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:41.998454 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:41.999354 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:41.999518 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:41.999613 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 06:46:41.999780 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14502,"thread_start_us":14365,"threads_started":1}
I20260809 06:46:42.002170 7585 tablet_service.cc:1467] Processing CreateTablet for tablet 880e3e0db6434a3faf66f8cc3a14c68b (DEFAULT_TABLE table=test-table [id=5778a62665c44b939e53dd5e3de147ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.002187 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 880e3e0db6434a3faf66f8cc3a14c68b (DEFAULT_TABLE table=test-table [id=5778a62665c44b939e53dd5e3de147ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.003086 7798 tablet_service.cc:1467] Processing CreateTablet for tablet 880e3e0db6434a3faf66f8cc3a14c68b (DEFAULT_TABLE table=test-table [id=5778a62665c44b939e53dd5e3de147ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.003676 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880e3e0db6434a3faf66f8cc3a14c68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.003688 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880e3e0db6434a3faf66f8cc3a14c68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.004206 7798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 880e3e0db6434a3faf66f8cc3a14c68b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.015369 8183 tablet_bootstrap.cc:492] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Bootstrap starting.
I20260809 06:46:42.017832 8188 tablet_bootstrap.cc:492] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Bootstrap starting.
I20260809 06:46:42.019194 8183 tablet_bootstrap.cc:654] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.019388 8190 tablet_bootstrap.cc:492] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Bootstrap starting.
I20260809 06:46:42.020447 8183 log.cc:824] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:42.022233 8188 tablet_bootstrap.cc:654] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.022698 8190 tablet_bootstrap.cc:654] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.022933 8183 tablet_bootstrap.cc:492] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: No bootstrap required, opened a new log
I20260809 06:46:42.023180 8183 ts_tablet_manager.cc:1397] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 06:46:42.023474 8188 log.cc:824] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:42.023867 8190 log.cc:824] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:42.026204 8188 tablet_bootstrap.cc:492] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: No bootstrap required, opened a new log
I20260809 06:46:42.026432 8188 ts_tablet_manager.cc:1397] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:46:42.026862 8190 tablet_bootstrap.cc:492] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: No bootstrap required, opened a new log
I20260809 06:46:42.027118 8190 ts_tablet_manager.cc:1397] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 06:46:42.033281 8183 raft_consensus.cc:348] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.033751 8183 raft_consensus.cc:374] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.033538 8188 raft_consensus.cc:348] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.033912 8188 raft_consensus.cc:374] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.033901 8183 raft_consensus.cc:729] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed8871ee87489784eb3153555979be, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.034009 8188 raft_consensus.cc:729] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.034622 8188 consensus_queue.cc:260] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.034554 8183 consensus_queue.cc:260] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.036248 7914 heartbeater.cc:503] Master 127.27.209.254:40897 was elected leader, sending a full tablet report...
I20260809 06:46:42.036849 8188 ts_tablet_manager.cc:1428] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260809 06:46:42.037091 8190 raft_consensus.cc:348] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.037536 8190 raft_consensus.cc:374] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.037681 8190 raft_consensus.cc:729] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7eb36a6cd410e8ba47f3f64a60f28, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.038280 8183 ts_tablet_manager.cc:1428] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260809 06:46:42.039201 8190 consensus_queue.cc:260] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.042389 8141 heartbeater.cc:503] Master 127.27.209.254:40897 was elected leader, sending a full tablet report...
I20260809 06:46:42.042923 8190 ts_tablet_manager.cc:1428] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260809 06:46:42.076963 7693 tablet.cc:2369] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:46:42.106354 7915 tablet.cc:2369] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:42.120067 8203 raft_consensus.cc:482] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:42.120398 8203 raft_consensus.cc:504] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.122092 8203 leader_election.cc:288] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691), 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911)
I20260809 06:46:42.128659 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.122600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:50971 (local address 127.27.209.195:33691)
0809 06:46:42.122821 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.122828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.123472 (+ 644us) server_negotiation.cc:407] Connection header received
0809 06:46:42.123679 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.123683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.123747 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.123821 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:42.124424 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.125174 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.126965 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.127753 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.127893 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.127969 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.128269 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.128390 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 06:46:42.128824 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.122336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33691 (local address 127.27.209.194:50971)
0809 06:46:42.123352 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.123371 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.123519 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.123849 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.123853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.123879 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:42.124281 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.124296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.125296 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.125301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.126816 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.126829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.127956 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.127962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.128125 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.128150 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.128231 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.128349 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":1,"thread_start_us":641,"threads_started":1}
W20260809 06:46:42.128670 8142 tablet.cc:2369] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:46:42.128877 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.122528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40911 (local address 127.27.209.194:41141)
0809 06:46:42.123494 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.123508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.123678 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.123965 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.123969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.124010 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:42.124405 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.124414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.125535 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.125540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.126967 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.126980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.128133 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.128139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.128267 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.128290 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.128370 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.128490 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":637,"threads_started":1}
I20260809 06:46:42.129078 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.122748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:41141 (local address 127.27.209.193:40911)
0809 06:46:42.122965 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.122972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.123615 (+ 643us) server_negotiation.cc:407] Connection header received
0809 06:46:42.123805 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.123809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.123864 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.123935 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:42.124558 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.125405 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.127123 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.127918 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.128057 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.128136 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.128739 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.128866 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:46:42.130146 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880e3e0db6434a3faf66f8cc3a14c68b" candidate_uuid: "6aed8871ee87489784eb3153555979be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4783a9e155fd41b3a72ef50eef248262" is_pre_election: true
I20260809 06:46:42.130131 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880e3e0db6434a3faf66f8cc3a14c68b" candidate_uuid: "6aed8871ee87489784eb3153555979be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" is_pre_election: true
I20260809 06:46:42.130687 7625 raft_consensus.cc:2454] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aed8871ee87489784eb3153555979be in term 0.
I20260809 06:46:42.130687 8070 raft_consensus.cc:2454] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aed8871ee87489784eb3153555979be in term 0.
I20260809 06:46:42.131418 7722 leader_election.cc:302] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:42.131831 8203 raft_consensus.cc:2781] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:42.131978 8203 raft_consensus.cc:482] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:42.132071 8203 raft_consensus.cc:3037] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.134202 8203 raft_consensus.cc:504] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.134760 8203 leader_election.cc:288] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [CANDIDATE]: Term 1 election: Requested vote from peers 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691), 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911)
I20260809 06:46:42.135244 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880e3e0db6434a3faf66f8cc3a14c68b" candidate_uuid: "6aed8871ee87489784eb3153555979be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4783a9e155fd41b3a72ef50eef248262"
I20260809 06:46:42.135244 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "880e3e0db6434a3faf66f8cc3a14c68b" candidate_uuid: "6aed8871ee87489784eb3153555979be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
I20260809 06:46:42.135447 7625 raft_consensus.cc:3037] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.135447 8070 raft_consensus.cc:3037] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.138619 8070 raft_consensus.cc:2454] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aed8871ee87489784eb3153555979be in term 1.
I20260809 06:46:42.138646 7625 raft_consensus.cc:2454] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aed8871ee87489784eb3153555979be in term 1.
I20260809 06:46:42.139020 7722 leader_election.cc:302] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ed7eb36a6cd410e8ba47f3f64a60f28, 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:42.139330 8203 raft_consensus.cc:2781] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:42.139782 8203 raft_consensus.cc:686] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 1 LEADER]: Becoming Leader. State: Replica: 6aed8871ee87489784eb3153555979be, State: Running, Role: LEADER
I20260809 06:46:42.140298 8203 consensus_queue.cc:237] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } }
I20260809 06:46:42.144459 7373 catalog_manager.cc:5495] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aed8871ee87489784eb3153555979be (127.27.209.194). New cstate: current_term: 1 leader_uuid: "6aed8871ee87489784eb3153555979be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } health_report { overall_health: UNKNOWN } } }
I20260809 06:46:42.190451 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.185376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.27.209.194:38947)
0809 06:46:42.185543 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.185548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.185565 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:46:42.185746 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.185750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.185857 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.185976 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:42.186281 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.187013 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.188413 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.188816 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.188904 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.189264 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:42.189878 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:42.189982 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.190077 (+ 95us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.190158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:46:42.198942 7625 raft_consensus.cc:1261] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Refusing update from remote peer 6aed8871ee87489784eb3153555979be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:42.198949 8070 raft_consensus.cc:1261] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Refusing update from remote peer 6aed8871ee87489784eb3153555979be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:42.199913 8209 consensus_queue.cc:1035] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [LEADER]: Connected to new peer: Peer: permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:42.200290 8203 consensus_queue.cc:1035] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:42.217516 8214 mvcc.cc:205] Tried to move back new op lower bound from 7316512776996409344 to 7316512776771084288. Current Snapshot: MvccSnapshot[applied={T|T < 7316512776996409344}]
I20260809 06:46:42.228945 7373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:42.229172 7373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:42.233654 7373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Sending DeleteTablet for 3 replicas of tablet 880e3e0db6434a3faf66f8cc3a14c68b
I20260809 06:46:42.234937 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 880e3e0db6434a3faf66f8cc3a14c68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:42 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56780
I20260809 06:46:42.235013 7585 tablet_service.cc:1514] Processing DeleteTablet for tablet 880e3e0db6434a3faf66f8cc3a14c68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:42 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42368
I20260809 06:46:42.235143 7798 tablet_service.cc:1514] Processing DeleteTablet for tablet 880e3e0db6434a3faf66f8cc3a14c68b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:42 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:41928
I20260809 06:46:42.237664 7373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:42.239189 7373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:42.248453 7583 tablet_service.cc:1467] Processing CreateTablet for tablet c525ecf8cc194d60abb459d2163a56ed (DEFAULT_TABLE table=test-table [id=34b2b495929c4c2abf6a154dd747ba46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.249101 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c525ecf8cc194d60abb459d2163a56ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.252350 8035 tablet_service.cc:1467] Processing CreateTablet for tablet c525ecf8cc194d60abb459d2163a56ed (DEFAULT_TABLE table=test-table [id=34b2b495929c4c2abf6a154dd747ba46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.252954 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c525ecf8cc194d60abb459d2163a56ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.256163 7796 tablet_service.cc:1467] Processing CreateTablet for tablet c525ecf8cc194d60abb459d2163a56ed (DEFAULT_TABLE table=test-table [id=34b2b495929c4c2abf6a154dd747ba46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.256785 7796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c525ecf8cc194d60abb459d2163a56ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.262630 8238 tablet_replica.cc:330] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: stopping tablet replica
I20260809 06:46:42.263154 8238 raft_consensus.cc:2229] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:42.264024 8238 raft_consensus.cc:2258] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:42.266320 8238 ts_tablet_manager.cc:1905] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:42.267565 8190 tablet_bootstrap.cc:492] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Bootstrap starting.
I20260809 06:46:42.269178 8190 tablet_bootstrap.cc:654] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.270008 8183 tablet_bootstrap.cc:492] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Bootstrap starting.
I20260809 06:46:42.271984 8235 tablet_replica.cc:330] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: stopping tablet replica
I20260809 06:46:42.272079 8183 tablet_bootstrap.cc:654] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.272480 8235 raft_consensus.cc:2229] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:42.273026 8235 raft_consensus.cc:2258] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:42.278074 8188 tablet_bootstrap.cc:492] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Bootstrap starting.
I20260809 06:46:42.279992 8188 tablet_bootstrap.cc:654] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.281798 8236 tablet_replica.cc:330] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: stopping tablet replica
I20260809 06:46:42.281910 8235 ts_tablet_manager.cc:1905] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:42.282311 8236 raft_consensus.cc:2229] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:42.283241 8236 raft_consensus.cc:2258] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:42.284898 8183 tablet_bootstrap.cc:492] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: No bootstrap required, opened a new log
I20260809 06:46:42.285053 8183 ts_tablet_manager.cc:1397] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260809 06:46:42.285835 8183 raft_consensus.cc:348] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.286060 8183 raft_consensus.cc:374] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.286165 8183 raft_consensus.cc:729] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed8871ee87489784eb3153555979be, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.286424 8183 consensus_queue.cc:260] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.286978 8183 ts_tablet_manager.cc:1428] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:46:42.288625 8190 tablet_bootstrap.cc:492] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: No bootstrap required, opened a new log
I20260809 06:46:42.288861 8190 ts_tablet_manager.cc:1397] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20260809 06:46:42.289884 8190 raft_consensus.cc:348] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.290144 8190 raft_consensus.cc:374] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.290282 8190 raft_consensus.cc:729] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7eb36a6cd410e8ba47f3f64a60f28, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.289533 8238 ts_tablet_manager.cc:1918] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:42.290623 8190 consensus_queue.cc:260] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.290920 8238 log.cc:1196] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/880e3e0db6434a3faf66f8cc3a14c68b
I20260809 06:46:42.291265 8190 ts_tablet_manager.cc:1428] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:46:42.291495 8238 ts_tablet_manager.cc:1939] T 880e3e0db6434a3faf66f8cc3a14c68b P 6aed8871ee87489784eb3153555979be: Deleting consensus metadata
I20260809 06:46:42.287422 8236 ts_tablet_manager.cc:1905] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:42.293311 7346 catalog_manager.cc:4841] TS 6aed8871ee87489784eb3153555979be (127.27.209.194:38947): tablet 880e3e0db6434a3faf66f8cc3a14c68b (table test-table [id=5778a62665c44b939e53dd5e3de147ba]) successfully deleted
I20260809 06:46:42.294178 8235 ts_tablet_manager.cc:1918] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:42.294405 8235 log.cc:1196] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/880e3e0db6434a3faf66f8cc3a14c68b
I20260809 06:46:42.295147 8235 ts_tablet_manager.cc:1939] T 880e3e0db6434a3faf66f8cc3a14c68b P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting consensus metadata
I20260809 06:46:42.297031 7346 catalog_manager.cc:4841] TS 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691): tablet 880e3e0db6434a3faf66f8cc3a14c68b (table test-table [id=5778a62665c44b939e53dd5e3de147ba]) successfully deleted
I20260809 06:46:42.297935 8236 ts_tablet_manager.cc:1918] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:42.298077 8236 log.cc:1196] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/880e3e0db6434a3faf66f8cc3a14c68b
I20260809 06:46:42.298470 8236 ts_tablet_manager.cc:1939] T 880e3e0db6434a3faf66f8cc3a14c68b P 4783a9e155fd41b3a72ef50eef248262: Deleting consensus metadata
I20260809 06:46:42.299871 7352 catalog_manager.cc:4841] TS 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911): tablet 880e3e0db6434a3faf66f8cc3a14c68b (table test-table [id=5778a62665c44b939e53dd5e3de147ba]) successfully deleted
I20260809 06:46:42.300485 8188 tablet_bootstrap.cc:492] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: No bootstrap required, opened a new log
I20260809 06:46:42.300616 8188 ts_tablet_manager.cc:1397] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.001s
I20260809 06:46:42.301203 8188 raft_consensus.cc:348] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.301370 8188 raft_consensus.cc:374] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.301462 8188 raft_consensus.cc:729] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.301720 8188 consensus_queue.cc:260] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.302139 8188 ts_tablet_manager.cc:1428] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260809 06:46:42.331123 8197 raft_consensus.cc:482] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:42.331358 8197 raft_consensus.cc:504] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.333232 8197 leader_election.cc:288] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aed8871ee87489784eb3153555979be (127.27.209.194:38947), 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:42.341572 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.333601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:56893 (local address 127.27.209.194:38947)
0809 06:46:42.333798 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.333806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.334085 (+ 279us) server_negotiation.cc:407] Connection header received
0809 06:46:42.334384 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.334390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.334501 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.334571 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:42.335375 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.336590 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.339642 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.340546 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.340695 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.340775 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.341126 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.341266 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:46:42.341665 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.333465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38947 (local address 127.27.209.193:56893)
0809 06:46:42.333984 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.333999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.334206 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.334802 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.334806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.334833 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:42.335162 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.335174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.336739 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.336744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.339208 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.339224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.340814 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.340823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.340955 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.340992 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.341087 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.341221 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":97,"thread_start_us":258,"threads_started":1}
I20260809 06:46:42.341830 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.334284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:58265 (local address 127.27.209.195:33691)
0809 06:46:42.334462 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.334468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.334857 (+ 389us) server_negotiation.cc:407] Connection header received
0809 06:46:42.334993 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.334997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.335074 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.335168 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:42.335762 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.336551 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.339366 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.340622 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.340779 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.340849 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.341411 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.341545 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 06:46:42.342496 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.334131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33691 (local address 127.27.209.193:58265)
0809 06:46:42.334763 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.334778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.334874 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.335209 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.335213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.335238 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:42.335588 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.335601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.336680 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.336685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.339216 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.339226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.341012 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.341019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.341217 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.341252 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.342008 (+ 756us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.342146 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":90,"thread_start_us":337,"threads_started":1}
I20260809 06:46:42.342615 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c525ecf8cc194d60abb459d2163a56ed" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be" is_pre_election: true
I20260809 06:46:42.343073 7840 raft_consensus.cc:2454] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 0.
I20260809 06:46:42.343693 7503 leader_election.cc:302] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:42.344058 8197 raft_consensus.cc:2781] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:42.344178 8197 raft_consensus.cc:482] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:42.344286 8197 raft_consensus.cc:3037] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.344709 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c525ecf8cc194d60abb459d2163a56ed" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" is_pre_election: true
I20260809 06:46:42.344949 8070 raft_consensus.cc:2454] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 0.
I20260809 06:46:42.346755 8197 raft_consensus.cc:504] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.347325 8197 leader_election.cc:288] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 election: Requested vote from peers 6aed8871ee87489784eb3153555979be (127.27.209.194:38947), 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:42.347848 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c525ecf8cc194d60abb459d2163a56ed" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
I20260809 06:46:42.348047 8070 raft_consensus.cc:3037] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.347999 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c525ecf8cc194d60abb459d2163a56ed" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be"
I20260809 06:46:42.348202 7840 raft_consensus.cc:3037] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.350545 7840 raft_consensus.cc:2454] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 1.
I20260809 06:46:42.350888 7503 leader_election.cc:302] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:42.350953 8070 raft_consensus.cc:2454] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 1.
I20260809 06:46:42.351123 8197 raft_consensus.cc:2781] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:42.351774 8197 raft_consensus.cc:686] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 1 LEADER]: Becoming Leader. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Running, Role: LEADER
I20260809 06:46:42.352416 8197 consensus_queue.cc:237] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.358583 7371 catalog_manager.cc:5495] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } health_report { overall_health: UNKNOWN } } }
Aug 09 06:46:42 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258000, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.254@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:42.421164 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.406069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.27.209.254:40897)
0809 06:46:42.406283 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.406290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.406868 (+ 578us) server_negotiation.cc:407] Connection header received
0809 06:46:42.407050 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.407054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.407113 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.407184 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:42.408730 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.409486 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.411282 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.411713 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.411804 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.415711 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:42.415715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:42.415719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:42.415756 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:42.418244 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:42.419125 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:42.419129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:42.419131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:42.419250 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:42.419641 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:42.419645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:42.419647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:42.419907 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:42.420032 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.420576 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.420857 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":74}
I20260809 06:46:42.421204 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.406065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:40897 (local address 127.0.0.1:37756)
0809 06:46:42.406756 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.406777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.406931 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.407250 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.407255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.408220 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:42.408576 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.408590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.409683 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.409690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.411155 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.411171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.411887 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.411894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.412076 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.412765 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:42.412789 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:42.415489 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:42.418368 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:42.418381 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:42.418390 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:42.419021 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:42.419346 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:42.419351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:42.419355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:42.419513 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:42.420035 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:42.420044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:42.420252 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.420433 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.420742 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":361,"threads_started":1}
I20260809 06:46:42.426556 7371 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:37756:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 06:46:42.427879 7371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:42.435235 7583 tablet_service.cc:1467] Processing CreateTablet for tablet b3a51bacadd044bdbfc49cb35abcdfc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=76f03dad2a2848af9ee2ef05b8987a5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:46:42.435329 7796 tablet_service.cc:1467] Processing CreateTablet for tablet b3a51bacadd044bdbfc49cb35abcdfc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=76f03dad2a2848af9ee2ef05b8987a5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:46:42.435850 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a51bacadd044bdbfc49cb35abcdfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.435902 7796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a51bacadd044bdbfc49cb35abcdfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.435745 8036 tablet_service.cc:1467] Processing CreateTablet for tablet b3a51bacadd044bdbfc49cb35abcdfc7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=76f03dad2a2848af9ee2ef05b8987a5b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 06:46:42.436348 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a51bacadd044bdbfc49cb35abcdfc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.443255 8183 tablet_bootstrap.cc:492] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: Bootstrap starting.
I20260809 06:46:42.443341 8188 tablet_bootstrap.cc:492] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: Bootstrap starting.
I20260809 06:46:42.444317 8190 tablet_bootstrap.cc:492] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Bootstrap starting.
I20260809 06:46:42.445019 8188 tablet_bootstrap.cc:654] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.445101 8183 tablet_bootstrap.cc:654] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.445923 8190 tablet_bootstrap.cc:654] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.454456 8183 tablet_bootstrap.cc:492] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: No bootstrap required, opened a new log
I20260809 06:46:42.454721 8183 ts_tablet_manager.cc:1397] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260809 06:46:42.455612 8183 raft_consensus.cc:348] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.455956 8183 raft_consensus.cc:374] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.456085 8183 raft_consensus.cc:729] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed8871ee87489784eb3153555979be, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.456419 8183 consensus_queue.cc:260] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.456979 8228 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.457363 8228 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.457366 8190 tablet_bootstrap.cc:492] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: No bootstrap required, opened a new log
I20260809 06:46:42.457544 8190 ts_tablet_manager.cc:1397] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260809 06:46:42.458037 8183 ts_tablet_manager.cc:1428] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 06:46:42.458369 8190 raft_consensus.cc:348] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.458662 8190 raft_consensus.cc:374] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.458818 8190 raft_consensus.cc:729] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7eb36a6cd410e8ba47f3f64a60f28, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.459169 8190 consensus_queue.cc:260] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.459712 8205 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.460047 8205 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.462455 8190 ts_tablet_manager.cc:1428] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260809 06:46:42.465175 8188 tablet_bootstrap.cc:492] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: No bootstrap required, opened a new log
I20260809 06:46:42.465355 8188 ts_tablet_manager.cc:1397] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.012s
I20260809 06:46:42.466001 8188 raft_consensus.cc:348] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.466218 8188 raft_consensus.cc:374] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.466300 8188 raft_consensus.cc:729] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.466548 8188 consensus_queue.cc:260] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.466933 8197 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.467154 8188 ts_tablet_manager.cc:1428] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 06:46:42.467250 8197 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.536914 8197 raft_consensus.cc:482] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:42.537133 8197 raft_consensus.cc:504] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.537789 8197 leader_election.cc:288] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aed8871ee87489784eb3153555979be (127.27.209.194:38947), 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:42.538357 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51bacadd044bdbfc49cb35abcdfc7" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be" is_pre_election: true
I20260809 06:46:42.538486 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51bacadd044bdbfc49cb35abcdfc7" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" is_pre_election: true
I20260809 06:46:42.538652 7840 raft_consensus.cc:2454] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 0.
I20260809 06:46:42.538828 8070 raft_consensus.cc:2454] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 0.
I20260809 06:46:42.539100 7503 leader_election.cc:302] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:42.539426 8197 raft_consensus.cc:2781] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:42.539572 8197 raft_consensus.cc:482] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:42.539719 8197 raft_consensus.cc:3037] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.542718 8197 raft_consensus.cc:504] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.543421 8197 leader_election.cc:288] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 election: Requested vote from peers 6aed8871ee87489784eb3153555979be (127.27.209.194:38947), 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:42.543866 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51bacadd044bdbfc49cb35abcdfc7" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
I20260809 06:46:42.543880 7840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a51bacadd044bdbfc49cb35abcdfc7" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be"
I20260809 06:46:42.544072 8070 raft_consensus.cc:3037] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.544147 7840 raft_consensus.cc:3037] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:42.546083 8070 raft_consensus.cc:2454] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 1.
I20260809 06:46:42.546468 7503 leader_election.cc:302] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed7eb36a6cd410e8ba47f3f64a60f28, 4783a9e155fd41b3a72ef50eef248262; no voters:
I20260809 06:46:42.546725 8197 raft_consensus.cc:2781] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:42.546875 8197 raft_consensus.cc:686] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [term 1 LEADER]: Becoming Leader. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Running, Role: LEADER
I20260809 06:46:42.547144 8197 consensus_queue.cc:237] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.547278 7840 raft_consensus.cc:2454] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 1.
I20260809 06:46:42.548987 8247 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: TxnStatusTablet state changed. Reason: New leader 4783a9e155fd41b3a72ef50eef248262. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.549423 8247 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:46:42.550273 8267 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 06:46:42.550544 8267 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 06:46:42.550875 7373 catalog_manager.cc:5495] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } health_report { overall_health: UNKNOWN } } }
I20260809 06:46:42.591030 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.583319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40911 (local address 127.0.0.1:42380)
0809 06:46:42.583607 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.583639 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.583786 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.584463 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.584469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.584483 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:46:42.584710 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.584721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.586260 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.586266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.587866 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.587883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.588857 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.588864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.589092 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.589178 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:46:42.590315 (+ 1137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:46:42.590341 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.590431 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.590608 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 06:46:42.591737 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.583825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.27.209.193:40911)
0809 06:46:42.584005 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.584011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.584031 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:46:42.584131 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.584137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.584239 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.584398 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:42.584880 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.586110 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.588046 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.588696 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.588823 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.589400 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:42.590193 (+ 793us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:42.590342 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.591273 (+ 931us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.591363 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:46:42.596342 8070 raft_consensus.cc:1261] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Refusing update from remote peer 4783a9e155fd41b3a72ef50eef248262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:42.597177 8197 consensus_queue.cc:1035] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:42.597833 7840 raft_consensus.cc:1261] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Refusing update from remote peer 4783a9e155fd41b3a72ef50eef248262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:42.599119 8197 consensus_queue.cc:1035] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:42.610237 8197 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.610545 8197 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:46:42.610589 8205 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: TxnStatusTablet state changed. Reason: New leader 4783a9e155fd41b3a72ef50eef248262. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.610848 8205 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.613603 8228 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: TxnStatusTablet state changed. Reason: New leader 4783a9e155fd41b3a72ef50eef248262. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.613798 8215 mvcc.cc:205] Tried to move back new op lower bound from 7316512778626813952 to 7316512778435866624. Current Snapshot: MvccSnapshot[applied={T|T < 7316512778626813952 or (T in {7316512778626813952})}]
I20260809 06:46:42.613925 8228 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.619735 8205 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.619975 8205 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 1ed7eb36a6cd410e8ba47f3f64a60f28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.626302 8247 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.626570 8247 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 4783a9e155fd41b3a72ef50eef248262: This TxnStatusTablet replica's current role is: LEADER
I20260809 06:46:42.628410 8213 mvcc.cc:205] Tried to move back new op lower bound from 7316512778626813952 to 7316512778435866624. Current Snapshot: MvccSnapshot[applied={T|T < 7316512778626813952 or (T in {7316512778626813952})}]
I20260809 06:46:42.629199 8228 tablet_replica.cc:439] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } } }
I20260809 06:46:42.629458 8228 tablet_replica.cc:442] T b3a51bacadd044bdbfc49cb35abcdfc7 P 6aed8871ee87489784eb3153555979be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 06:46:42.632998 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.620827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.27.209.193:40911)
0809 06:46:42.621138 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.621144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.621163 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:46:42.621226 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.621231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.621309 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.622069 (+ 760us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:42.622397 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.623513 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.625763 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.631420 (+ 5657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.631620 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.632160 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:42.632306 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:42.632411 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.632574 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.632709 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260809 06:46:42.652827 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.645383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:43645 (local address 127.27.209.193:40911)
0809 06:46:42.645605 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.645612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.646220 (+ 608us) server_negotiation.cc:407] Connection header received
0809 06:46:42.646445 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.646451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.646529 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.646626 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:42.647264 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.648460 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.650640 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.651296 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.651416 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.652001 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:42.652145 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:42.652236 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.652401 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.652485 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 06:46:42.652956 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.645082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40911 (local address 127.27.209.193:43645)
0809 06:46:42.646111 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:42.646131 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:42.646307 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:42.646679 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:42.646684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:42.646697 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 06:46:42.647097 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.647112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.648602 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.648608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.650468 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:42.650485 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.651498 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.651504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:42.651742 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.651821 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 06:46:42.652262 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 06:46:42.652275 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 06:46:42.652365 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 06:46:42.652505 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":686,"threads_started":1}
I20260809 06:46:42.665912 8070 raft_consensus.cc:1261] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Refusing update from remote peer 4783a9e155fd41b3a72ef50eef248262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:42.666045 7840 raft_consensus.cc:1261] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Refusing update from remote peer 4783a9e155fd41b3a72ef50eef248262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:42.666743 8197 consensus_queue.cc:1035] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:42.667222 8247 consensus_queue.cc:1035] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:42.759408 7373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:42.759732 7373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:42.762459 7373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Sending DeleteTablet for 3 replicas of tablet c525ecf8cc194d60abb459d2163a56ed
I20260809 06:46:42.763366 7796 tablet_service.cc:1514] Processing DeleteTablet for tablet c525ecf8cc194d60abb459d2163a56ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:42 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:41928
I20260809 06:46:42.763518 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet c525ecf8cc194d60abb459d2163a56ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:42 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42368
I20260809 06:46:42.763482 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet c525ecf8cc194d60abb459d2163a56ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:42 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56780
I20260809 06:46:42.763657 8238 tablet_replica.cc:330] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: stopping tablet replica
I20260809 06:46:42.763792 8235 tablet_replica.cc:330] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: stopping tablet replica
I20260809 06:46:42.763850 8238 raft_consensus.cc:2229] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:42.763937 8235 raft_consensus.cc:2229] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:42.764024 8238 raft_consensus.cc:2258] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:42.764068 8235 raft_consensus.cc:2258] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:42.764277 8236 tablet_replica.cc:330] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: stopping tablet replica
I20260809 06:46:42.764595 8236 raft_consensus.cc:2229] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:42.765080 8236 raft_consensus.cc:2258] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:42.765538 8235 ts_tablet_manager.cc:1905] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:42.766456 8238 ts_tablet_manager.cc:1905] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:42.771291 8236 ts_tablet_manager.cc:1905] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:42.772166 8235 ts_tablet_manager.cc:1918] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:46:42.772374 8235 log.cc:1196] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/c525ecf8cc194d60abb459d2163a56ed
I20260809 06:46:42.772857 8235 ts_tablet_manager.cc:1939] T c525ecf8cc194d60abb459d2163a56ed P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting consensus metadata
I20260809 06:46:42.774003 7346 catalog_manager.cc:4841] TS 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691): tablet c525ecf8cc194d60abb459d2163a56ed (table test-table [id=34b2b495929c4c2abf6a154dd747ba46]) successfully deleted
I20260809 06:46:42.778221 8236 ts_tablet_manager.cc:1918] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:46:42.778461 8236 log.cc:1196] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/c525ecf8cc194d60abb459d2163a56ed
I20260809 06:46:42.778609 8238 ts_tablet_manager.cc:1918] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:46:42.778833 8238 log.cc:1196] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/c525ecf8cc194d60abb459d2163a56ed
I20260809 06:46:42.778972 8236 ts_tablet_manager.cc:1939] T c525ecf8cc194d60abb459d2163a56ed P 4783a9e155fd41b3a72ef50eef248262: Deleting consensus metadata
I20260809 06:46:42.779269 8238 ts_tablet_manager.cc:1939] T c525ecf8cc194d60abb459d2163a56ed P 6aed8871ee87489784eb3153555979be: Deleting consensus metadata
I20260809 06:46:42.780256 7352 catalog_manager.cc:4841] TS 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911): tablet c525ecf8cc194d60abb459d2163a56ed (table test-table [id=34b2b495929c4c2abf6a154dd747ba46]) successfully deleted
I20260809 06:46:42.780256 7346 catalog_manager.cc:4841] TS 6aed8871ee87489784eb3153555979be (127.27.209.194:38947): tablet c525ecf8cc194d60abb459d2163a56ed (table test-table [id=34b2b495929c4c2abf6a154dd747ba46]) successfully deleted
Aug 09 06:46:42 dist-test-slave-jpj1 krb5kdc[7303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 06:46:42 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:42 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:46:42.839923 7516 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:42.825048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.27.209.254:40897)
0809 06:46:42.825340 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.825347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.825880 (+ 533us) server_negotiation.cc:407] Connection header received
0809 06:46:42.826060 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.826065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.826134 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.826221 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:42.827456 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.828211 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.829485 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.829940 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.830052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.839632 (+ 9580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37760: BlockingRecv error: recv got EOF from 127.0.0.1:37760 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Aug 09 06:46:42 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM
I20260809 06:46:42.857118 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:42.843427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.27.209.254:40897)
0809 06:46:42.843629 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:42.843635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:42.844094 (+ 459us) server_negotiation.cc:407] Connection header received
0809 06:46:42.844309 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:42.844314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:42.844374 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:42.844452 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:42.845715 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.846472 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.847677 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:42.848079 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:42.848185 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:42.851908 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:42.851913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:42.851916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:42.851950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:42.854409 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:42.855090 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:42.855096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:42.855099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:42.855246 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:42.855741 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:42.855745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:42.855747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:42.856024 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:42.856269 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:42.856557 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 06:46:42.856772 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:46:42.860486 7373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:42.861119 7373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:42.867705 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 76c522d3a0434f7695a326d294fb65c3 (DEFAULT_TABLE table=test-table [id=a1b9e5208fd749e5b4601350dde39909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.868264 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c522d3a0434f7695a326d294fb65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.868219 8036 tablet_service.cc:1467] Processing CreateTablet for tablet 76c522d3a0434f7695a326d294fb65c3 (DEFAULT_TABLE table=test-table [id=a1b9e5208fd749e5b4601350dde39909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.868619 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c522d3a0434f7695a326d294fb65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.871402 7796 tablet_service.cc:1467] Processing CreateTablet for tablet 76c522d3a0434f7695a326d294fb65c3 (DEFAULT_TABLE table=test-table [id=a1b9e5208fd749e5b4601350dde39909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:42.871882 7796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c522d3a0434f7695a326d294fb65c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:42.874922 8190 tablet_bootstrap.cc:492] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Bootstrap starting.
I20260809 06:46:42.875054 8188 tablet_bootstrap.cc:492] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Bootstrap starting.
I20260809 06:46:42.876760 8188 tablet_bootstrap.cc:654] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.877028 8190 tablet_bootstrap.cc:654] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.879773 8183 tablet_bootstrap.cc:492] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Bootstrap starting.
I20260809 06:46:42.881376 8183 tablet_bootstrap.cc:654] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:42.888777 8190 tablet_bootstrap.cc:492] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: No bootstrap required, opened a new log
I20260809 06:46:42.889029 8190 ts_tablet_manager.cc:1397] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260809 06:46:42.889964 8190 raft_consensus.cc:348] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.890383 8190 raft_consensus.cc:374] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.890525 8190 raft_consensus.cc:729] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7eb36a6cd410e8ba47f3f64a60f28, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.890918 8190 consensus_queue.cc:260] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.891748 8190 ts_tablet_manager.cc:1428] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260809 06:46:42.900974 8183 tablet_bootstrap.cc:492] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: No bootstrap required, opened a new log
I20260809 06:46:42.901196 8183 ts_tablet_manager.cc:1397] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.004s
I20260809 06:46:42.901366 8188 tablet_bootstrap.cc:492] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: No bootstrap required, opened a new log
I20260809 06:46:42.901603 8188 ts_tablet_manager.cc:1397] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.000s
I20260809 06:46:42.901963 8183 raft_consensus.cc:348] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.902267 8183 raft_consensus.cc:374] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.902390 8183 raft_consensus.cc:729] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed8871ee87489784eb3153555979be, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.902526 8188 raft_consensus.cc:348] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.902679 8183 consensus_queue.cc:260] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.902853 8188 raft_consensus.cc:374] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:42.902995 8188 raft_consensus.cc:729] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Initialized, Role: FOLLOWER
I20260809 06:46:42.903328 8188 consensus_queue.cc:260] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:42.903955 8188 ts_tablet_manager.cc:1428] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 06:46:42.905488 8183 ts_tablet_manager.cc:1428] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260809 06:46:43.001468 8205 raft_consensus.cc:482] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:43.001667 8205 raft_consensus.cc:504] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.003196 8205 leader_election.cc:288] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911), 6aed8871ee87489784eb3153555979be (127.27.209.194:38947)
I20260809 06:46:43.011102 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.003680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:34421 (local address 127.27.209.193:40911)
0809 06:46:43.003900 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.003907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.005062 (+ 1155us) server_negotiation.cc:407] Connection header received
0809 06:46:43.005323 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.005328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.005424 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.005490 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:43.006210 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.007195 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.008999 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.010154 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.010324 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.010397 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.010724 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.010857 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:46:43.011480 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.003489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38947 (local address 127.27.209.195:41355)
0809 06:46:43.004261 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:43.004285 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:43.004430 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:43.005204 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:43.005209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:43.005231 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:43.005522 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:43.005535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.007056 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.007062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:43.009209 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:43.009224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.010692 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.010698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:43.010835 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.010857 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:46:43.010935 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:46:43.011059 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":25,"thread_start_us":800,"threads_started":1}
I20260809 06:46:43.011804 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.003516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40911 (local address 127.27.209.195:34421)
0809 06:46:43.004967 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:43.004985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:43.005163 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:43.005688 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:43.005691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:43.005715 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:46:43.006042 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:43.006054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.007314 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.007318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:43.008866 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:43.008878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.010370 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.010375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:43.010522 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.010553 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:46:43.011287 (+ 734us) client_negotiation.cc:240] Negotiation successful
0809 06:46:43.011413 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":29,"thread_start_us":465,"threads_started":1}
I20260809 06:46:43.011933 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.003903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:41355 (local address 127.27.209.194:38947)
0809 06:46:43.004882 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.004887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.004906 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:46:43.004983 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.004988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.005065 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.005682 (+ 617us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:46:43.005848 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.006902 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.009376 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.010491 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.010634 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.010701 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.010972 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.011100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":718,"threads_started":1}
I20260809 06:46:43.012558 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c522d3a0434f7695a326d294fb65c3" candidate_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be" is_pre_election: true
I20260809 06:46:43.012589 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c522d3a0434f7695a326d294fb65c3" candidate_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4783a9e155fd41b3a72ef50eef248262" is_pre_election: true
I20260809 06:46:43.012804 7839 raft_consensus.cc:2454] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed7eb36a6cd410e8ba47f3f64a60f28 in term 0.
I20260809 06:46:43.012840 7625 raft_consensus.cc:2454] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed7eb36a6cd410e8ba47f3f64a60f28 in term 0.
I20260809 06:46:43.013319 7948 leader_election.cc:302] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ed7eb36a6cd410e8ba47f3f64a60f28, 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:43.013761 8205 raft_consensus.cc:2781] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:43.013890 8205 raft_consensus.cc:482] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:43.014004 8205 raft_consensus.cc:3037] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:43.016180 8205 raft_consensus.cc:504] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.016811 8205 leader_election.cc:288] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [CANDIDATE]: Term 1 election: Requested vote from peers 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911), 6aed8871ee87489784eb3153555979be (127.27.209.194:38947)
I20260809 06:46:43.017253 7625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c522d3a0434f7695a326d294fb65c3" candidate_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4783a9e155fd41b3a72ef50eef248262"
I20260809 06:46:43.017356 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c522d3a0434f7695a326d294fb65c3" candidate_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be"
I20260809 06:46:43.017489 7625 raft_consensus.cc:3037] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:43.017556 7839 raft_consensus.cc:3037] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:43.020462 7839 raft_consensus.cc:2454] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed7eb36a6cd410e8ba47f3f64a60f28 in term 1.
I20260809 06:46:43.020540 7625 raft_consensus.cc:2454] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed7eb36a6cd410e8ba47f3f64a60f28 in term 1.
I20260809 06:46:43.020877 7948 leader_election.cc:302] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed7eb36a6cd410e8ba47f3f64a60f28, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:43.021178 8205 raft_consensus.cc:2781] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:43.021600 8205 raft_consensus.cc:686] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed7eb36a6cd410e8ba47f3f64a60f28, State: Running, Role: LEADER
I20260809 06:46:43.021981 8205 consensus_queue.cc:237] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.025513 7369 catalog_manager.cc:5495] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195). New cstate: current_term: 1 leader_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } health_report { overall_health: HEALTHY } } }
I20260809 06:46:43.047797 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.041011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.27.209.195:33691)
0809 06:46:43.041530 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.041534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.041553 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:46:43.041635 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.041638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.041713 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.041875 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:43.042283 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.043061 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.045054 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.045557 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.045671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.046311 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:43.047112 (+ 801us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:43.047244 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.047366 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.047435 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":280,"threads_started":1}
I20260809 06:46:43.051836 7839 raft_consensus.cc:1261] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Refusing update from remote peer 1ed7eb36a6cd410e8ba47f3f64a60f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:43.051872 7625 raft_consensus.cc:1261] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Refusing update from remote peer 1ed7eb36a6cd410e8ba47f3f64a60f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:43.052568 8319 consensus_queue.cc:1035] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:43.052937 8205 consensus_queue.cc:1035] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:43.076107 7369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:43.076393 7369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:43.080251 7369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Sending DeleteTablet for 3 replicas of tablet 76c522d3a0434f7695a326d294fb65c3
I20260809 06:46:43.081058 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c522d3a0434f7695a326d294fb65c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:43 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42368
I20260809 06:46:43.081288 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c522d3a0434f7695a326d294fb65c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:43 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56780
I20260809 06:46:43.081387 8236 tablet_replica.cc:330] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: stopping tablet replica
I20260809 06:46:43.081575 8235 tablet_replica.cc:330] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: stopping tablet replica
I20260809 06:46:43.081600 8236 raft_consensus.cc:2229] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:43.081749 8235 raft_consensus.cc:2229] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:43.081862 8236 raft_consensus.cc:2258] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:43.082582 7796 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c522d3a0434f7695a326d294fb65c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:43 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:41928
I20260809 06:46:43.082883 8238 tablet_replica.cc:330] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: stopping tablet replica
I20260809 06:46:43.083046 8238 raft_consensus.cc:2229] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:43.083292 8238 raft_consensus.cc:2258] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:43.083499 8235 raft_consensus.cc:2258] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:43.083829 8236 ts_tablet_manager.cc:1905] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:43.084455 8238 ts_tablet_manager.cc:1905] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:43.088553 8235 ts_tablet_manager.cc:1905] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:43.090657 8238 ts_tablet_manager.cc:1918] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:43.090778 8238 log.cc:1196] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/76c522d3a0434f7695a326d294fb65c3
I20260809 06:46:43.091254 8238 ts_tablet_manager.cc:1939] T 76c522d3a0434f7695a326d294fb65c3 P 6aed8871ee87489784eb3153555979be: Deleting consensus metadata
I20260809 06:46:43.092425 7346 catalog_manager.cc:4841] TS 6aed8871ee87489784eb3153555979be (127.27.209.194:38947): tablet 76c522d3a0434f7695a326d294fb65c3 (table test-table [id=a1b9e5208fd749e5b4601350dde39909]) successfully deleted
I20260809 06:46:43.092486 7369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:46:43.093070 7369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:46:43.097666 8235 ts_tablet_manager.cc:1918] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:43.097787 8235 log.cc:1196] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/76c522d3a0434f7695a326d294fb65c3
I20260809 06:46:43.102468 7583 tablet_service.cc:1467] Processing CreateTablet for tablet ce4f6e297ed94876b38b6a6089b6b752 (DEFAULT_TABLE table=test-table [id=271c4f99df5a4e57b35dd0421d073143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:43.102530 7796 tablet_service.cc:1467] Processing CreateTablet for tablet ce4f6e297ed94876b38b6a6089b6b752 (DEFAULT_TABLE table=test-table [id=271c4f99df5a4e57b35dd0421d073143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:43.102927 7796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce4f6e297ed94876b38b6a6089b6b752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:43.102926 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce4f6e297ed94876b38b6a6089b6b752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:43.103346 8235 ts_tablet_manager.cc:1939] T 76c522d3a0434f7695a326d294fb65c3 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting consensus metadata
I20260809 06:46:43.104422 7346 catalog_manager.cc:4841] TS 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691): tablet 76c522d3a0434f7695a326d294fb65c3 (table test-table [id=a1b9e5208fd749e5b4601350dde39909]) successfully deleted
I20260809 06:46:43.107896 8036 tablet_service.cc:1467] Processing CreateTablet for tablet ce4f6e297ed94876b38b6a6089b6b752 (DEFAULT_TABLE table=test-table [id=271c4f99df5a4e57b35dd0421d073143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:46:43.108318 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce4f6e297ed94876b38b6a6089b6b752. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:43.110388 8188 tablet_bootstrap.cc:492] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Bootstrap starting.
I20260809 06:46:43.112115 8188 tablet_bootstrap.cc:654] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:43.114658 8183 tablet_bootstrap.cc:492] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Bootstrap starting.
I20260809 06:46:43.116222 8183 tablet_bootstrap.cc:654] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:43.118707 8190 tablet_bootstrap.cc:492] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Bootstrap starting.
I20260809 06:46:43.119083 8236 ts_tablet_manager.cc:1918] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:46:43.119205 8236 log.cc:1196] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/76c522d3a0434f7695a326d294fb65c3
I20260809 06:46:43.120357 8236 ts_tablet_manager.cc:1939] T 76c522d3a0434f7695a326d294fb65c3 P 4783a9e155fd41b3a72ef50eef248262: Deleting consensus metadata
I20260809 06:46:43.120411 8190 tablet_bootstrap.cc:654] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:43.122820 7352 catalog_manager.cc:4841] TS 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911): tablet 76c522d3a0434f7695a326d294fb65c3 (table test-table [id=a1b9e5208fd749e5b4601350dde39909]) successfully deleted
I20260809 06:46:43.139477 8190 tablet_bootstrap.cc:492] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: No bootstrap required, opened a new log
I20260809 06:46:43.139820 8190 ts_tablet_manager.cc:1397] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.005s
I20260809 06:46:43.140805 8190 raft_consensus.cc:348] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.141597 8190 raft_consensus.cc:374] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:43.141748 8190 raft_consensus.cc:729] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed7eb36a6cd410e8ba47f3f64a60f28, State: Initialized, Role: FOLLOWER
I20260809 06:46:43.142112 8190 consensus_queue.cc:260] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.143813 8190 ts_tablet_manager.cc:1428] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 06:46:43.145617 8183 tablet_bootstrap.cc:492] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: No bootstrap required, opened a new log
I20260809 06:46:43.145859 8183 ts_tablet_manager.cc:1397] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.005s
I20260809 06:46:43.146721 8183 raft_consensus.cc:348] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.147042 8183 raft_consensus.cc:374] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:43.147190 8183 raft_consensus.cc:729] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aed8871ee87489784eb3153555979be, State: Initialized, Role: FOLLOWER
I20260809 06:46:43.147523 8183 consensus_queue.cc:260] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.149431 8188 tablet_bootstrap.cc:492] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: No bootstrap required, opened a new log
I20260809 06:46:43.149712 8188 ts_tablet_manager.cc:1397] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.006s
I20260809 06:46:43.150177 8183 ts_tablet_manager.cc:1428] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 06:46:43.150570 8188 raft_consensus.cc:348] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.150916 8188 raft_consensus.cc:374] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:43.151060 8188 raft_consensus.cc:729] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Initialized, Role: FOLLOWER
I20260809 06:46:43.151501 8188 consensus_queue.cc:260] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.152590 8188 ts_tablet_manager.cc:1428] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 06:46:43.323045 8247 raft_consensus.cc:482] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:46:43.323274 8247 raft_consensus.cc:504] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.324079 8247 leader_election.cc:288] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aed8871ee87489784eb3153555979be (127.27.209.194:38947), 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:43.324817 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4f6e297ed94876b38b6a6089b6b752" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be" is_pre_election: true
I20260809 06:46:43.324829 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4f6e297ed94876b38b6a6089b6b752" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" is_pre_election: true
I20260809 06:46:43.325083 7839 raft_consensus.cc:2454] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 0.
I20260809 06:46:43.325114 8070 raft_consensus.cc:2454] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 0.
I20260809 06:46:43.325662 7503 leader_election.cc:302] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4783a9e155fd41b3a72ef50eef248262, 6aed8871ee87489784eb3153555979be; no voters:
I20260809 06:46:43.326210 8247 raft_consensus.cc:2781] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:46:43.326331 8247 raft_consensus.cc:482] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:46:43.326444 8247 raft_consensus.cc:3037] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:43.329847 8247 raft_consensus.cc:504] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.330562 8247 leader_election.cc:288] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 election: Requested vote from peers 6aed8871ee87489784eb3153555979be (127.27.209.194:38947), 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691)
I20260809 06:46:43.331139 7839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4f6e297ed94876b38b6a6089b6b752" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aed8871ee87489784eb3153555979be"
I20260809 06:46:43.331209 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce4f6e297ed94876b38b6a6089b6b752" candidate_uuid: "4783a9e155fd41b3a72ef50eef248262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28"
I20260809 06:46:43.331383 7839 raft_consensus.cc:3037] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:43.331427 8070 raft_consensus.cc:3037] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:43.334563 7839 raft_consensus.cc:2454] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 1.
I20260809 06:46:43.334599 8070 raft_consensus.cc:2454] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4783a9e155fd41b3a72ef50eef248262 in term 1.
I20260809 06:46:43.335143 7503 leader_election.cc:302] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed7eb36a6cd410e8ba47f3f64a60f28, 4783a9e155fd41b3a72ef50eef248262; no voters:
I20260809 06:46:43.335454 8247 raft_consensus.cc:2781] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:43.335996 8247 raft_consensus.cc:686] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 1 LEADER]: Becoming Leader. State: Replica: 4783a9e155fd41b3a72ef50eef248262, State: Running, Role: LEADER
I20260809 06:46:43.336337 8247 consensus_queue.cc:237] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } }
I20260809 06:46:43.341689 7369 catalog_manager.cc:5495] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "4783a9e155fd41b3a72ef50eef248262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4783a9e155fd41b3a72ef50eef248262" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 } health_report { overall_health: UNKNOWN } } }
I20260809 06:46:43.426393 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.419377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.27.209.193:40911)
0809 06:46:43.419594 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.419601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.419908 (+ 307us) server_negotiation.cc:407] Connection header received
0809 06:46:43.420125 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.420131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.420218 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.420321 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:46:43.420788 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.421964 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.423888 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.424590 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.424745 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.425271 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:46:43.425437 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:46:43.425533 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.425789 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.425875 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:46:43.439368 7839 raft_consensus.cc:1261] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Refusing update from remote peer 4783a9e155fd41b3a72ef50eef248262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:43.439556 8070 raft_consensus.cc:1261] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Refusing update from remote peer 4783a9e155fd41b3a72ef50eef248262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:46:43.440229 8336 consensus_queue.cc:1035] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aed8871ee87489784eb3153555979be" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:43.440726 8247 consensus_queue.cc:1035] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed7eb36a6cd410e8ba47f3f64a60f28" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:46:43.530228 7373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:43.530498 7373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:46:43.533969 7373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42990dade52942dd9db38a0b7c48c0cf: Sending DeleteTablet for 3 replicas of tablet ce4f6e297ed94876b38b6a6089b6b752
I20260809 06:46:43.534783 7585 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4f6e297ed94876b38b6a6089b6b752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:43 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42368
I20260809 06:46:43.534849 7796 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4f6e297ed94876b38b6a6089b6b752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:43 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:41928
I20260809 06:46:43.535089 8236 tablet_replica.cc:330] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: stopping tablet replica
I20260809 06:46:43.535116 8238 tablet_replica.cc:330] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: stopping tablet replica
I20260809 06:46:43.535301 8238 raft_consensus.cc:2229] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:43.535312 8236 raft_consensus.cc:2229] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:46:43.535501 8238 raft_consensus.cc:2258] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:43.535645 8236 raft_consensus.cc:2258] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:43.536666 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet ce4f6e297ed94876b38b6a6089b6b752 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:46:43 UTC) from {username='oryx', principal='oryx/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56780
I20260809 06:46:43.537060 8235 tablet_replica.cc:330] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: stopping tablet replica
I20260809 06:46:43.537268 8235 raft_consensus.cc:2229] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:46:43.537454 8235 raft_consensus.cc:2258] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:46:43.537722 8236 ts_tablet_manager.cc:1905] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:43.538340 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 7479
I20260809 06:46:43.544142 8235 ts_tablet_manager.cc:1905] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:46:43.545378 8236 ts_tablet_manager.cc:1918] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:46:43.545533 8236 log.cc:1196] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/ce4f6e297ed94876b38b6a6089b6b752
I20260809 06:46:43.546111 8236 ts_tablet_manager.cc:1939] T ce4f6e297ed94876b38b6a6089b6b752 P 4783a9e155fd41b3a72ef50eef248262: Deleting consensus metadata
I20260809 06:46:43.552655 8235 ts_tablet_manager.cc:1918] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:46:43.553175 8235 log.cc:1196] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/ce4f6e297ed94876b38b6a6089b6b752
I20260809 06:46:43.553687 8235 ts_tablet_manager.cc:1939] T ce4f6e297ed94876b38b6a6089b6b752 P 1ed7eb36a6cd410e8ba47f3f64a60f28: Deleting consensus metadata
I20260809 06:46:43.553678 7352 catalog_manager.cc:4841] TS 4783a9e155fd41b3a72ef50eef248262 (127.27.209.193:40911): tablet ce4f6e297ed94876b38b6a6089b6b752 (table test-table [id=271c4f99df5a4e57b35dd0421d073143]) successfully deleted
I20260809 06:46:43.554971 7346 catalog_manager.cc:4841] TS 1ed7eb36a6cd410e8ba47f3f64a60f28 (127.27.209.195:33691): tablet ce4f6e297ed94876b38b6a6089b6b752 (table test-table [id=271c4f99df5a4e57b35dd0421d073143]) successfully deleted
I20260809 06:46:43.556098 8238 ts_tablet_manager.cc:1905] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:46:43 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.193@KRBTEST.COM
I20260809 06:46:43.563248 8238 ts_tablet_manager.cc:1918] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 06:46:43.563877 8238 log.cc:1196] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/ce4f6e297ed94876b38b6a6089b6b752
I20260809 06:46:43.564334 8238 ts_tablet_manager.cc:1939] T ce4f6e297ed94876b38b6a6089b6b752 P 6aed8871ee87489784eb3153555979be: Deleting consensus metadata
I20260809 06:46:43.565723 7346 catalog_manager.cc:4841] TS 6aed8871ee87489784eb3153555979be (127.27.209.194:38947): tablet ce4f6e297ed94876b38b6a6089b6b752 (table test-table [id=271c4f99df5a4e57b35dd0421d073143]) successfully deleted
I20260809 06:46:43.570536 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.540038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.27.209.193:40911)
0809 06:46:43.540277 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.540283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.547712 (+ 7429us) server_negotiation.cc:407] Connection header received
0809 06:46:43.548011 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.548018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.550319 (+ 2301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.550408 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:43.552328 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.553431 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.555907 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.556583 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.556687 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.562488 (+ 5801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:43.562505 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:43.562526 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:43.562566 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:43.566766 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:43.567670 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:43.567680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:43.567684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:43.567919 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:43.568406 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:43.568411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:43.568414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:43.568888 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:43.569245 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.569933 (+ 688us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.570171 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260809 06:46:43.571924 7683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:43.714270 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 7479
I20260809 06:46:43.745572 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 7697
Aug 09 06:46:43 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.194@KRBTEST.COM
I20260809 06:46:43.760386 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.746345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.27.209.194:38947)
0809 06:46:43.746777 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.746782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.746799 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:46:43.746857 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.746861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.746927 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.747017 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:43.748269 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.748953 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.750143 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.750565 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.750658 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.754674 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:43.754689 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:43.754708 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:43.754745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:43.757645 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:43.758247 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:43.758257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:43.758260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:43.758397 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:43.758731 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:43.758736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:43.758740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:43.759092 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:43.759368 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.759896 (+ 528us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.760104 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260809 06:46:43.760983 7905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:43.882278 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 7697
I20260809 06:46:43.914745 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 7923
Aug 09 06:46:43 dist-test-slave-jpj1 krb5kdc[7303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.209.195@KRBTEST.COM
I20260809 06:46:43.929126 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:43.915457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.27.209.195:33691)
0809 06:46:43.915945 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:43.915949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:43.915966 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:46:43.916017 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:43.916021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:43.916086 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:43.916173 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:43.917408 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.918106 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.919278 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:43.919723 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:43.919825 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:43.923894 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:43.923910 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:43.923929 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:43.923960 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:43.926784 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:43.927367 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:43.927374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:43.927376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:43.927512 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:43.927861 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:43.927865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:43.927867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:43.928199 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:43.928448 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:43.928701 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 06:46:43.928868 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
I20260809 06:46:43.929775 8131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:44.054070 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 7923
I20260809 06:46:44.078969 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 7321
I20260809 06:46:44.080188 7432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:44.195319 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 7321
2026-08-09T06:46:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5858 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:44 dist-test-slave-jpj1 krb5kdc[8355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:44 dist-test-slave-jpj1 krb5kdc[8355](info): set up 2 sockets
Aug 09 06:46:44 dist-test-slave-jpj1 krb5kdc[8355](info): commencing operation
krb5kdc: starting...
W20260809 06:46:45.388677 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:45 dist-test-slave-jpj1 krb5kdc[8355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:46:45.620656 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:46821
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:46821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:45.943888 8375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:45.944176 8375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:45.944324 8375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:45.951361 8375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:45.951483 8375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:45.951591 8375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:45.951678 8375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:45.951756 8375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:45.961593 8375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:46821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:46821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:45.962924 8375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:45.964349 8375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:45.976043 8383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:45.976007 8384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:45.975965 8388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:45.976795 8375 server_base.cc:1034] running on GCE node
I20260809 06:46:45.977629 8375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:45.979076 8375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:45.980300 8375 hybrid_clock.cc:648] HybridClock initialized: now 1786258005980226 us; error 64 us; skew 500 ppm
I20260809 06:46:45.980681 8375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T06:46:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1856 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:46:46 dist-test-slave-jpj1 krb5kdc[8396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:46:46 dist-test-slave-jpj1 krb5kdc[8396](info): set up 2 sockets
Aug 09 06:46:46 dist-test-slave-jpj1 krb5kdc[8396](info): commencing operation
krb5kdc: starting...
W20260809 06:46:47.261894 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:46:47 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:46:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:46:47.513312 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36411
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:47.836407 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:47.836683 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:47.836824 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:47.843777 8414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:47.843885 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:47.843966 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:47.844041 8414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:47.844100 8414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:47.853539 8414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:47.854760 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:47.856096 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:47.867218 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:47.867250 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:47.867280 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:47.868132 8414 server_base.cc:1034] running on GCE node
I20260809 06:46:47.868894 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:47.870216 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:47.871415 8414 hybrid_clock.cc:648] HybridClock initialized: now 1786258007871366 us; error 34 us; skew 500 ppm
I20260809 06:46:47.871769 8414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:47 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258007, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:47.878504 8414 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:47.879947 8414 webserver.cc:460] Webserver started at http://127.27.209.254:34975/ using document root <none> and password file <none>
I20260809 06:46:47.880594 8414 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:47.880710 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:47.881013 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:47.883927 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "d8f1ebc8a323479d87a151be9407abd7"
format_stamp: "Formatted at 2026-08-09 06:46:47 on dist-test-slave-jpj1"
server_key: "92e1e30e26ae8a78d430bbc345773025"
server_key_iv: "374fb9041f937dcca919725bb20a932a"
server_key_version: "encryptionkey@0"
I20260809 06:46:47.884656 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "d8f1ebc8a323479d87a151be9407abd7"
format_stamp: "Formatted at 2026-08-09 06:46:47 on dist-test-slave-jpj1"
server_key: "92e1e30e26ae8a78d430bbc345773025"
server_key_iv: "374fb9041f937dcca919725bb20a932a"
server_key_version: "encryptionkey@0"
I20260809 06:46:47.889539 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 06:46:47.892879 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:47.894104 8414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:46:47.894258 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "d8f1ebc8a323479d87a151be9407abd7"
format_stamp: "Formatted at 2026-08-09 06:46:47 on dist-test-slave-jpj1"
server_key: "92e1e30e26ae8a78d430bbc345773025"
server_key_iv: "374fb9041f937dcca919725bb20a932a"
server_key_version: "encryptionkey@0"
I20260809 06:46:47.894440 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:47.908421 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:47.912153 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:47.912410 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:47.929375 8414 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:36411
I20260809 06:46:47.929399 8524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:36411 every 8 connection(s)
I20260809 06:46:47.930716 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:47.934403 8525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:46:47.935209 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 8414
I20260809 06:46:47.935396 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:46:47.936061 28487 external_mini_cluster.cc:1428] Setting key b8cbc9240c84a052fe1a91e96f5d1a0f
I20260809 06:46:47.948000 8525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Bootstrap starting.
Aug 09 06:46:47 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:47.951557 8525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Neither blocks nor log segments found. Creating new log.
I20260809 06:46:47.952728 8525 log.cc:824] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:47.955466 8525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: No bootstrap required, opened a new log
I20260809 06:46:47.958235 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:47.938011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.27.209.254:36411)
0809 06:46:47.938794 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:47.938805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:47.938846 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:46:47.940475 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:47.940497 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:47.940873 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:47.941184 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:47.943016 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:47.944376 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:47.946120 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:47.946742 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:47.946879 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:47.951336 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:47.951345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:47.951358 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:47.951393 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:47.954348 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:47.954975 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:47.954981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:47.954987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:47.955147 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:47.955519 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:47.955524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:47.955527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:47.956082 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:47.956361 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:47.956826 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 06:46:47.957231 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260809 06:46:47.963129 8525 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:47.963616 8525 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:46:47.963716 8525 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f1ebc8a323479d87a151be9407abd7, State: Initialized, Role: FOLLOWER
I20260809 06:46:47.964285 8525 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:47.964505 8525 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:47.964635 8525 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:47.964778 8525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:46:47.966806 8525 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:47.967255 8525 leader_election.cc:302] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8f1ebc8a323479d87a151be9407abd7; no voters:
I20260809 06:46:47.967815 8525 leader_election.cc:288] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:46:47.967926 8533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:46:47.968603 8533 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 LEADER]: Becoming Leader. State: Replica: d8f1ebc8a323479d87a151be9407abd7, State: Running, Role: LEADER
I20260809 06:46:47.969058 8533 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:47.969883 8525 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:47.977284 8534 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8f1ebc8a323479d87a151be9407abd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } } }
I20260809 06:46:47.977195 8535 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8f1ebc8a323479d87a151be9407abd7. Latest consensus state: current_term: 1 leader_uuid: "d8f1ebc8a323479d87a151be9407abd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } } }
I20260809 06:46:47.977591 8535 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:47.977591 8534 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:47.983742 8555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:47.988731 8555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:47.997282 8555 catalog_manager.cc:1347] Generated new cluster ID: e3fac0271af44e5bb223d103f7461100
I20260809 06:46:47.997408 8555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:48.007193 8555 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:46:48.008530 8555 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:48.023878 8555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Generated new TSK 0
I20260809 06:46:48.025000 8555 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:46:48.082857 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36411
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:48.412652 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:48.412951 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:48.413079 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:48.420519 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:48.420622 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:48.420774 8565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:48.430404 8565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:48.431668 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:48.432966 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:48.445300 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:48.445340 8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:48.445393 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:48.446393 8565 server_base.cc:1034] running on GCE node
I20260809 06:46:48.446943 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:48.447834 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:48.449055 8565 hybrid_clock.cc:648] HybridClock initialized: now 1786258008449007 us; error 35 us; skew 500 ppm
I20260809 06:46:48.449370 8565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:48 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:48.452384 8565 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:46:48.453714 8565 webserver.cc:460] Webserver started at http://127.27.209.193:36625/ using document root <none> and password file <none>
I20260809 06:46:48.454350 8565 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:48.454465 8565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:48.454754 8565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:48.457762 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "e6472a4b222b412a8a1ca14460f0a0e9"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "27401c1e6590a1243079cfe942e3ce1e"
server_key_iv: "1ca377520e0edc19bee6efe58c581619"
server_key_version: "encryptionkey@0"
I20260809 06:46:48.458523 8565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "e6472a4b222b412a8a1ca14460f0a0e9"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "27401c1e6590a1243079cfe942e3ce1e"
server_key_iv: "1ca377520e0edc19bee6efe58c581619"
server_key_version: "encryptionkey@0"
I20260809 06:46:48.463445 8565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:46:48.466786 8596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:48.468133 8565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:46:48.468274 8565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "e6472a4b222b412a8a1ca14460f0a0e9"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "27401c1e6590a1243079cfe942e3ce1e"
server_key_iv: "1ca377520e0edc19bee6efe58c581619"
server_key_version: "encryptionkey@0"
I20260809 06:46:48.468447 8565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:48.500243 8565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:48.504140 8565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:48.504393 8565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:48.505348 8565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:48.506660 8565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:48.506738 8565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:48.506865 8565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:48.506917 8565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:48.536844 8565 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:33293
I20260809 06:46:48.536921 8785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:33293 every 8 connection(s)
I20260809 06:46:48.538074 8565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:48.547422 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 8565
I20260809 06:46:48.547652 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:46:48.548372 28487 external_mini_cluster.cc:1428] Setting key 0d6a36344fba8b0e1a53e5c368c9e434
Aug 09 06:46:48 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:48.558197 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:48.540499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:56933 (local address 127.27.209.254:36411)
0809 06:46:48.540961 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:48.540966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:48.541872 (+ 906us) server_negotiation.cc:407] Connection header received
0809 06:46:48.543710 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:48.543716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:48.543809 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:48.543907 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:48.545423 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:48.546165 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:48.548119 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:48.548583 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:48.548725 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:48.552641 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:48.552646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:48.552650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:48.552681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:48.555169 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:48.555930 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:48.555934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:48.555937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:48.556060 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:48.556482 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:48.556485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:48.556487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:48.556761 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:48.556894 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:48.557612 (+ 718us) server_negotiation.cc:299] Negotiation successful
0809 06:46:48.557834 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":231,"threads_started":1}
I20260809 06:46:48.559299 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:48.541041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:56933)
0809 06:46:48.541711 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:48.541754 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:48.543410 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:48.544170 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:48.544178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:48.544702 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:48.545257 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:48.545276 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:48.546312 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:48.546315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:48.547949 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:48.547965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:48.548847 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:48.548855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:48.549094 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:48.549827 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:48.549849 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:48.552405 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:48.555303 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:48.555312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:48.555329 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:48.555804 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:48.556167 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:48.556172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:48.556175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:48.556364 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:48.556885 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:48.556893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:48.557224 (+ 331us) client_negotiation.cc:769] Sending connection context
0809 06:46:48.557466 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 06:46:48.557982 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":265,"threads_started":1}
I20260809 06:46:48.560897 8787 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36411
I20260809 06:46:48.561189 8787 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:48.561887 8787 heartbeater.cc:511] Master 127.27.209.254:36411 requested a full tablet report, sending...
I20260809 06:46:48.564144 8469 ts_manager.cc:194] Registered new tserver with Master: e6472a4b222b412a8a1ca14460f0a0e9 (127.27.209.193:33293)
I20260809 06:46:48.566123 8469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:56933
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:46:48.600858 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36411
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:48.921834 8795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:48.922094 8795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:48.922192 8795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:48.929127 8795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:48.929225 8795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:48.929381 8795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:48.938705 8795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:48.939860 8795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:48.941156 8795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:48.953619 8801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:48.953523 8800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:48.953495 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:48.954203 8795 server_base.cc:1034] running on GCE node
I20260809 06:46:48.954772 8795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:48.955660 8795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:48.956861 8795 hybrid_clock.cc:648] HybridClock initialized: now 1786258008956816 us; error 33 us; skew 500 ppm
I20260809 06:46:48.957161 8795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:48 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:48.960079 8795 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:46:48.961454 8795 webserver.cc:460] Webserver started at http://127.27.209.194:37661/ using document root <none> and password file <none>
I20260809 06:46:48.962116 8795 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:48.962231 8795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:48.962523 8795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:48.965368 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "d88d9d5413ca4b1495974f91108213fa"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "acd1fd422a8cca89f78f31e272410fb2"
server_key_iv: "de6750ff351a7e09e5a73df9a4a02fa4"
server_key_version: "encryptionkey@0"
I20260809 06:46:48.966167 8795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "d88d9d5413ca4b1495974f91108213fa"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "acd1fd422a8cca89f78f31e272410fb2"
server_key_iv: "de6750ff351a7e09e5a73df9a4a02fa4"
server_key_version: "encryptionkey@0"
I20260809 06:46:48.971177 8795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:46:48.974793 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:48.976284 8795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:48.976440 8795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "d88d9d5413ca4b1495974f91108213fa"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "acd1fd422a8cca89f78f31e272410fb2"
server_key_iv: "de6750ff351a7e09e5a73df9a4a02fa4"
server_key_version: "encryptionkey@0"
I20260809 06:46:48.976630 8795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:49.008150 8795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:49.012046 8795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:49.012319 8795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:49.013267 8795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:49.014603 8795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:49.014680 8795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:49.014803 8795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:49.014855 8795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:49.052778 8795 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:39293
I20260809 06:46:49.052824 9010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:39293 every 8 connection(s)
I20260809 06:46:49.054030 8795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:49.062680 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 8795
I20260809 06:46:49.062880 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:46:49.063446 28487 external_mini_cluster.cc:1428] Setting key 86fbd76800a6e0a3dda51bc8586b2598
Aug 09 06:46:49 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:49.074198 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.056387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:48067 (local address 127.27.209.254:36411)
0809 06:46:49.056553 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:49.056558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:49.057677 (+ 1119us) server_negotiation.cc:407] Connection header received
0809 06:46:49.059408 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:49.059413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:49.059491 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:49.059629 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:49.061191 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.061889 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.063991 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.064568 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.064683 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.068385 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:49.068391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:49.068396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:49.068430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:49.071011 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.071847 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.071853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.071856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.071994 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.072461 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.072466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.072469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.072747 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:49.072880 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:49.073629 (+ 749us) server_negotiation.cc:299] Negotiation successful
0809 06:46:49.073857 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:46:49.075376 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.056815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:48067)
0809 06:46:49.057497 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:49.057542 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:49.059132 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:49.059823 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:49.059830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:49.060487 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:49.061037 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:49.061053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.062010 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.062013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:49.063861 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:49.063876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.064798 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.064806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:49.065088 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.065807 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:49.065831 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:49.068171 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:49.071142 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:49.071151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:49.071169 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:49.071734 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:49.072119 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:49.072124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:49.072127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:49.072321 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:49.072869 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:49.072876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:49.073243 (+ 367us) client_negotiation.cc:769] Sending connection context
0809 06:46:49.073510 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 06:46:49.074043 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260809 06:46:49.077203 9014 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36411
I20260809 06:46:49.077498 9014 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:49.078210 9014 heartbeater.cc:511] Master 127.27.209.254:36411 requested a full tablet report, sending...
I20260809 06:46:49.079708 8469 ts_manager.cc:194] Registered new tserver with Master: d88d9d5413ca4b1495974f91108213fa (127.27.209.194:39293)
I20260809 06:46:49.080593 8469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:48067
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:46:49.118709 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36411
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:49.448650 9023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:49.448939 9023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:49.449040 9023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:49.456306 9023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:49.456418 9023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:49.456537 9023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:49.466339 9023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:49.467578 9023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:49.468897 9023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:49.481889 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:49.481927 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:49.481889 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:49.482791 9023 server_base.cc:1034] running on GCE node
I20260809 06:46:49.483372 9023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:49.484294 9023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:49.485499 9023 hybrid_clock.cc:648] HybridClock initialized: now 1786258009485455 us; error 34 us; skew 500 ppm
I20260809 06:46:49.485812 9023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:49 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:49.488801 9023 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:46:49.490157 9023 webserver.cc:460] Webserver started at http://127.27.209.195:35781/ using document root <none> and password file <none>
I20260809 06:46:49.490805 9023 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:49.490914 9023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:49.491205 9023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:46:49.494047 9023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "8dbc70ec1658408dbb8eaf9db98929a7"
format_stamp: "Formatted at 2026-08-09 06:46:49 on dist-test-slave-jpj1"
server_key: "6359b7f2dd909863f42d7669c939b2bf"
server_key_iv: "0a6e0a8bb23e124f01491c6a13d4033f"
server_key_version: "encryptionkey@0"
I20260809 06:46:49.494800 9023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "8dbc70ec1658408dbb8eaf9db98929a7"
format_stamp: "Formatted at 2026-08-09 06:46:49 on dist-test-slave-jpj1"
server_key: "6359b7f2dd909863f42d7669c939b2bf"
server_key_iv: "0a6e0a8bb23e124f01491c6a13d4033f"
server_key_version: "encryptionkey@0"
I20260809 06:46:49.499853 9023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 06:46:49.503113 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:49.504421 9023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:49.504570 9023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8dbc70ec1658408dbb8eaf9db98929a7"
format_stamp: "Formatted at 2026-08-09 06:46:49 on dist-test-slave-jpj1"
server_key: "6359b7f2dd909863f42d7669c939b2bf"
server_key_iv: "0a6e0a8bb23e124f01491c6a13d4033f"
server_key_version: "encryptionkey@0"
I20260809 06:46:49.504777 9023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:49.530278 9023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:49.534222 9023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:49.534492 9023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:49.535418 9023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:49.536882 9023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:49.536964 9023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:49.537082 9023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:49.537135 9023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:49.566494 9023 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:33607
I20260809 06:46:49.566530 9227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:33607 every 8 connection(s)
I20260809 06:46:49.567793 9023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:49.569332 8787 heartbeater.cc:503] Master 127.27.209.254:36411 was elected leader, sending a full tablet report...
I20260809 06:46:49.571899 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 9023
I20260809 06:46:49.572067 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:46:49.572578 28487 external_mini_cluster.cc:1428] Setting key 49739dd8f7bab249de075c43e3139895
Aug 09 06:46:49 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:46:49.586858 8790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.570250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:59657 (local address 127.27.209.254:36411)
0809 06:46:49.570418 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:49.570422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:49.571926 (+ 1504us) server_negotiation.cc:407] Connection header received
0809 06:46:49.573958 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:49.573964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:49.574045 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:49.574176 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:49.575891 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.576739 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.578246 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.578657 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.578746 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.581962 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:49.581966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:49.581970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:49.582001 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:49.584287 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.584895 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.584899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.584901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.585026 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.585348 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.585351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.585353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.585592 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:49.585735 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:49.586393 (+ 658us) server_negotiation.cc:299] Negotiation successful
0809 06:46:49.586592 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:46:49.587920 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.570843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:59657)
0809 06:46:49.571748 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:49.571811 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:49.573649 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:49.574403 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:49.574415 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:49.574988 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:49.575707 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:49.575733 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.576935 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.576939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:49.578131 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:49.578143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.578828 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.578833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:49.579017 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.579649 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:49.579669 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:49.581765 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:46:49.584397 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:49.584403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:49.584415 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:49.584808 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:49.585098 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:46:49.585101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:46:49.585103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:46:49.585268 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:46:49.585695 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:46:49.585700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:46:49.586022 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 06:46:49.586263 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 06:46:49.586729 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":355,"threads_started":1}
I20260809 06:46:49.589160 9228 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36411
I20260809 06:46:49.589428 9228 heartbeater.cc:464] Registering TS with master...
I20260809 06:46:49.590095 9228 heartbeater.cc:511] Master 127.27.209.254:36411 requested a full tablet report, sending...
I20260809 06:46:49.591424 8469 ts_manager.cc:194] Registered new tserver with Master: 8dbc70ec1658408dbb8eaf9db98929a7 (127.27.209.195:33607)
I20260809 06:46:49.592188 8469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:59657
I20260809 06:46:49.599570 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab.
Entry for principal oryx/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab.
WARNING: no policy specified for oryx/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab.
Entry for principal oryx/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab.
WARNING: no policy specified for oryx/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab.
Entry for principal oryx/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab.
I20260809 06:46:49.735427 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 8565
Aug 09 06:46:49 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:46:49.752245 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.736714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.27.209.193:33293)
0809 06:46:49.737834 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:49.737841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:49.737867 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:46:49.737927 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:49.737934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:49.738175 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:49.738320 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:49.739862 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.740930 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.742255 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.742838 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.742967 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.746925 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:49.746936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:49.746951 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:49.746990 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:49.749524 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.750194 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.750201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.750203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.750344 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.750705 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.750709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.750711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.751043 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:49.751352 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:49.751639 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 06:46:49.751908 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":797,"threads_started":1}
I20260809 06:46:49.753594 8776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:49.834054 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 8565
I20260809 06:46:49.855008 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 8795
Aug 09 06:46:49 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:46:49.869889 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.855865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.27.209.194:39293)
0809 06:46:49.856398 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:49.856406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:49.856434 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:46:49.856504 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:49.856510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:49.856709 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:49.856860 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:49.857962 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.858983 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.860266 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.860813 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.860908 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.864798 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:49.864809 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:49.864823 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:49.864855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:49.867520 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.868140 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.868146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.868149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.868286 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.868561 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.868563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.868565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.868877 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:49.869091 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:49.869362 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 06:46:49.869565 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":260,"threads_started":1}
I20260809 06:46:49.871119 8988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:49.955250 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 8795
I20260809 06:46:49.977226 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9023
Aug 09 06:46:49 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:46:49.992854 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:49.978273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.27.209.195:33607)
0809 06:46:49.978965 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:49.978972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:49.979001 (+ 29us) server_negotiation.cc:407] Connection header received
0809 06:46:49.979062 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:49.979069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:49.979273 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:49.979396 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:49.980683 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.981687 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.982881 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:49.983437 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:49.983572 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:49.987737 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:49.987750 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:49.987765 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:49.987799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:49.990421 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.991027 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.991034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.991036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.991170 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:49.991512 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:49.991516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:49.991518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:49.991915 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:49.992171 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:49.992350 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 06:46:49.992557 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":368,"threads_started":1}
I20260809 06:46:49.994066 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:50.077934 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 9023
I20260809 06:46:50.102007 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 8414
I20260809 06:46:50.103255 8517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:46:50.206517 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 8414
I20260809 06:46:50.229233 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36411
--webserver_interface=127.27.209.254
--webserver_port=34975
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:46:50.556260 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:50.556543 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:50.556659 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:50.563753 9260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:46:50.563849 9260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:50.563920 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:50.563994 9260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:46:50.564054 9260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:46:50.573633 9260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:36411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=34975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:50.574851 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:50.576174 9260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:50.587406 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:50.587365 9272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:50.587435 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:50.588338 9260 server_base.cc:1034] running on GCE node
I20260809 06:46:50.589037 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:50.590364 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:50.591586 9260 hybrid_clock.cc:648] HybridClock initialized: now 1786258010591501 us; error 51 us; skew 500 ppm
I20260809 06:46:50.591924 9260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:50 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258010, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:50.594899 9260 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:46:50.596302 9260 webserver.cc:460] Webserver started at http://127.27.209.254:34975/ using document root <none> and password file <none>
I20260809 06:46:50.596951 9260 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:50.597081 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:50.602233 9260 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 06:46:50.604907 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:50.606120 9260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 06:46:50.606276 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "d8f1ebc8a323479d87a151be9407abd7"
format_stamp: "Formatted at 2026-08-09 06:46:47 on dist-test-slave-jpj1"
server_key: "92e1e30e26ae8a78d430bbc345773025"
server_key_iv: "374fb9041f937dcca919725bb20a932a"
server_key_version: "encryptionkey@0"
I20260809 06:46:50.607214 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:50.645390 9260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:50.649540 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:50.649796 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:50.667068 9260 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:36411
I20260809 06:46:50.667088 9354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:36411 every 8 connection(s)
I20260809 06:46:50.668491 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:46:50.672112 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 9260
I20260809 06:46:50.673070 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:33293
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:36411
--webserver_port=36625
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:46:50.676611 9357 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:46:50.680836 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Bootstrap starting.
I20260809 06:46:50.699852 9357 log.cc:824] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:46:50.708566 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:46:50.709053 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Bootstrap complete.
I20260809 06:46:50.718029 9357 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:50.718796 9357 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f1ebc8a323479d87a151be9407abd7, State: Initialized, Role: FOLLOWER
I20260809 06:46:50.719373 9357 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:50.719620 9357 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:46:50.719731 9357 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:46:50.719907 9357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:46:50.723580 9357 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:50.724061 9357 leader_election.cc:302] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8f1ebc8a323479d87a151be9407abd7; no voters:
I20260809 06:46:50.724637 9357 leader_election.cc:288] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:46:50.724781 9361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:46:50.725936 9361 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [term 2 LEADER]: Becoming Leader. State: Replica: d8f1ebc8a323479d87a151be9407abd7, State: Running, Role: LEADER
I20260809 06:46:50.726567 9361 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } }
I20260809 06:46:50.726743 9357 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:46:50.729009 9366 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8f1ebc8a323479d87a151be9407abd7. Latest consensus state: current_term: 2 leader_uuid: "d8f1ebc8a323479d87a151be9407abd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } } }
I20260809 06:46:50.729363 9366 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:50.729887 9364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d8f1ebc8a323479d87a151be9407abd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f1ebc8a323479d87a151be9407abd7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36411 } } }
I20260809 06:46:50.730170 9364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:46:50.735224 9373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:46:50.741318 9373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:46:50.742966 9373 catalog_manager.cc:1259] Loaded cluster ID: e3fac0271af44e5bb223d103f7461100
I20260809 06:46:50.743124 9373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:46:50.747277 9373 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:46:50.748848 9373 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8f1ebc8a323479d87a151be9407abd7: Loaded TSK: 0
I20260809 06:46:50.750563 9373 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:46:51.036497 9359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:51.036773 9359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:51.036901 9359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:51.044052 9359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:51.044175 9359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:51.044322 9359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:46:51.053972 9359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.193.keytab
--principal=oryx/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:33293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=36625
--tserver_master_addrs=127.27.209.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:51.055279 9359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:51.056681 9359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:51.071343 9396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:51.071341 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:51.072264 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:51.072527 9359 server_base.cc:1034] running on GCE node
I20260809 06:46:51.073161 9359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:51.074184 9359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:51.075385 9359 hybrid_clock.cc:648] HybridClock initialized: now 1786258011075321 us; error 61 us; skew 500 ppm
I20260809 06:46:51.075739 9359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258011, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:51.078945 9359 init.cc:377] Logged in from keytab as oryx/127.27.209.193@KRBTEST.COM (short username oryx)
I20260809 06:46:51.080554 9359 webserver.cc:460] Webserver started at http://127.27.209.193:36625/ using document root <none> and password file <none>
I20260809 06:46:51.081244 9359 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:51.081379 9359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:51.086715 9359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 06:46:51.089393 9411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.090728 9359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:46:51.090876 9359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "e6472a4b222b412a8a1ca14460f0a0e9"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "27401c1e6590a1243079cfe942e3ce1e"
server_key_iv: "1ca377520e0edc19bee6efe58c581619"
server_key_version: "encryptionkey@0"
I20260809 06:46:51.091835 9359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:51.110767 9359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:51.115044 9359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:51.115303 9359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:51.116403 9359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:51.117878 9359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:51.117965 9359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.118072 9359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:51.118144 9359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.148277 9359 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:33293
I20260809 06:46:51.148356 9602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:33293 every 8 connection(s)
I20260809 06:46:51.149725 9359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:46:51.159328 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 9359
I20260809 06:46:51.160082 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:39293
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:36411
--webserver_port=37661
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:51.168224 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:51.152702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:60161)
0809 06:46:51.153710 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:51.153750 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:51.155407 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:51.156814 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:51.156825 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:51.157445 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:51.157974 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.157992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.159630 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.159636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.161456 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.161472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.162542 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.162548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.162814 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.163490 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:51.163522 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:51.167228 (+ 3706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":539,"threads_started":1}
W20260809 06:46:51.168764 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:51.152676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:60161 (local address 127.27.209.254:36411)
0809 06:46:51.153492 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:51.153508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:51.153880 (+ 372us) server_negotiation.cc:407] Connection header received
0809 06:46:51.155966 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:51.155979 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:51.156299 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:51.156619 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:51.158157 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.159438 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.161597 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.162297 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.162436 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.167675 (+ 5239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:60161: BlockingRecv error: recv got EOF from 127.27.209.193:60161 (error 108)
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":278,"threads_started":1}
W20260809 06:46:51.169042 9603 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:36411 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.209.254:36411: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:51.177481 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:51.169616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:37843)
0809 06:46:51.169791 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:51.169806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:51.169886 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:51.170192 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:51.170197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:51.170585 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:51.170760 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.170770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.171750 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.171755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.173058 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.173073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.173710 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.173715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.173867 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.174391 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:51.174414 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:51.177280 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 06:46:51.177754 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:51.169694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:37843 (local address 127.27.209.254:36411)
0809 06:46:51.169847 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:51.169852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:51.169870 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:46:51.169982 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:51.169986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:51.170059 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:51.170135 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:51.170875 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.171606 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.173176 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.173569 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.173654 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.177546 (+ 3892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:37843: BlockingRecv error: recv got EOF from 127.27.209.193:37843 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:51.187139 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:51.178082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:42121)
0809 06:46:51.178686 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:51.178704 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:51.178824 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:51.179118 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:51.179124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:51.179657 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:51.179911 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.179922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.180913 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.180918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.182283 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.182300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.183120 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.183126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.183301 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.183877 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:51.183901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:51.186886 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":277,"threads_started":1}
W20260809 06:46:51.187422 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:51.178335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:42121 (local address 127.27.209.254:36411)
0809 06:46:51.178498 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:51.178503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:51.178779 (+ 276us) server_negotiation.cc:407] Connection header received
0809 06:46:51.178913 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:51.178916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:51.178980 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:51.179052 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:51.180039 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.180764 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.182412 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.182941 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.183050 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.187141 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:42121: BlockingRecv error: recv got EOF from 127.27.209.193:42121 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:46:51.187439 9603 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:46:51.501760 9608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:51.502017 9608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:51.502138 9608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:51.509032 9608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:51.509128 9608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:51.509277 9608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:46:51.518628 9608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.194.keytab
--principal=oryx/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:39293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=37661
--tserver_master_addrs=127.27.209.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:51.519917 9608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:51.521176 9608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:51.533638 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:51.533676 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:51.533715 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:51.534469 9608 server_base.cc:1034] running on GCE node
I20260809 06:46:51.535036 9608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:51.536029 9608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:51.537214 9608 hybrid_clock.cc:648] HybridClock initialized: now 1786258011537159 us; error 49 us; skew 500 ppm
I20260809 06:46:51.537510 9608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258011, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:51.540509 9608 init.cc:377] Logged in from keytab as oryx/127.27.209.194@KRBTEST.COM (short username oryx)
I20260809 06:46:51.541895 9608 webserver.cc:460] Webserver started at http://127.27.209.194:37661/ using document root <none> and password file <none>
I20260809 06:46:51.542546 9608 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:51.542681 9608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:51.547966 9608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:46:51.550513 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.551777 9608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:46:51.551921 9608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "d88d9d5413ca4b1495974f91108213fa"
format_stamp: "Formatted at 2026-08-09 06:46:48 on dist-test-slave-jpj1"
server_key: "acd1fd422a8cca89f78f31e272410fb2"
server_key_iv: "de6750ff351a7e09e5a73df9a4a02fa4"
server_key_version: "encryptionkey@0"
I20260809 06:46:51.552848 9608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:51.567765 9608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:51.571755 9608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:51.572003 9608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:51.572883 9608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:51.574155 9608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:51.574250 9608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.574355 9608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:51.574438 9608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.604578 9608 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:39293
I20260809 06:46:51.604637 9809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:39293 every 8 connection(s)
I20260809 06:46:51.606006 9608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:46:51.613821 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 9608
I20260809 06:46:51.614387 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:33607
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:36411
--webserver_port=35781
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:46:51.621284 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:51.608484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:51021 (local address 127.27.209.254:36411)
0809 06:46:51.608701 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:51.608706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:51.609918 (+ 1212us) server_negotiation.cc:407] Connection header received
0809 06:46:51.611795 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:51.611801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:51.611891 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:51.611975 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:51.613447 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.614213 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.615828 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.616261 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.616371 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.621042 (+ 4671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:51021: BlockingRecv error: recv got EOF from 127.27.209.194:51021 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 06:46:51.621948 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:51.608957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:51021)
0809 06:46:51.609757 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:51.609796 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:51.611488 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:51.612216 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:51.612224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:51.612743 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:51.613287 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.613304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.614364 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.614368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.615661 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.615682 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.616481 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.616489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.616727 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.617335 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:51.617357 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:51.620853 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":330,"threads_started":1}
W20260809 06:46:51.622531 9810 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:36411 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.209.254:36411: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:51.631269 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:51.623079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:60165)
0809 06:46:51.623265 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:51.623282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:51.623362 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:51.623687 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:51.623690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:51.623965 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:51.624094 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.624100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.625188 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.625194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.626477 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.626492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.627168 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.627173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.627341 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.627979 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:51.628001 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:51.630975 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 06:46:51.631393 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:51.623219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:60165 (local address 127.27.209.254:36411)
0809 06:46:51.623395 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:51.623401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:51.623419 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:46:51.623467 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:51.623472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:51.623564 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:51.623658 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:51.624231 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.625035 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.626597 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.627025 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.627129 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.631177 (+ 4048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:60165: BlockingRecv error: recv got EOF from 127.27.209.194:60165 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:51.639613 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:51.632047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:45685)
0809 06:46:51.632268 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:51.632281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:51.632361 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:51.632605 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:51.632608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:51.632866 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:51.632995 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.633002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.634232 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.634236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.635179 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:51.635193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.636100 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.636104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:51.636245 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.636727 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:51.636743 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:51.639309 (+ 2566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260809 06:46:51.639638 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:51.632168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:45685 (local address 127.27.209.254:36411)
0809 06:46:51.632323 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:51.632328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:51.632345 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:46:51.632436 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:51.632440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:51.632500 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:51.632603 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:51.633121 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.634120 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.635321 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:51.635971 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:51.636097 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:51.639452 (+ 3355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:45685: BlockingRecv error: recv got EOF from 127.27.209.194:45685 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 06:46:51.639986 9810 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:46:51.944294 9816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:46:51.944574 9816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:46:51.944675 9816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:46:51.951740 9816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:46:51.951850 9816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:46:51.951970 9816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:46:51.961489 9816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/krb5kdc/oryx_127.27.209.195.keytab
--principal=oryx/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:33607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=35781
--tserver_master_addrs=127.27.209.254:36411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:46:51.962708 9816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:46:51.964017 9816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:46:51.976472 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:51.976385 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:46:51.976437 9828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:46:51.977216 9816 server_base.cc:1034] running on GCE node
I20260809 06:46:51.977800 9816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:46:51.978670 9816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:46:51.979864 9816 hybrid_clock.cc:648] HybridClock initialized: now 1786258011979811 us; error 44 us; skew 500 ppm
I20260809 06:46:51.980166 9816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:46:51 dist-test-slave-jpj1 krb5kdc[8396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258011, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:46:51.983098 9816 init.cc:377] Logged in from keytab as oryx/127.27.209.195@KRBTEST.COM (short username oryx)
I20260809 06:46:51.984526 9816 webserver.cc:460] Webserver started at http://127.27.209.195:35781/ using document root <none> and password file <none>
I20260809 06:46:51.985205 9816 fs_manager.cc:359] Metadata directory not provided
I20260809 06:46:51.985338 9816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:46:51.990651 9816 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:46:51.993299 9837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:51.994581 9816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:46:51.994737 9816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8dbc70ec1658408dbb8eaf9db98929a7"
format_stamp: "Formatted at 2026-08-09 06:46:49 on dist-test-slave-jpj1"
server_key: "6359b7f2dd909863f42d7669c939b2bf"
server_key_iv: "0a6e0a8bb23e124f01491c6a13d4033f"
server_key_version: "encryptionkey@0"
I20260809 06:46:51.995693 9816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:46:52.014698 9816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:46:52.018474 9816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:46:52.018738 9816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:46:52.019716 9816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:46:52.021050 9816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:46:52.021147 9816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:52.021261 9816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:46:52.021332 9816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:46:52.051739 9816 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:33607
I20260809 06:46:52.051792 10021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:33607 every 8 connection(s)
I20260809 06:46:52.053146 9816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:46:52.060680 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 9816
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
W20260809 06:46:52.070130 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:52.055577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:48741 (local address 127.27.209.254:36411)
0809 06:46:52.055755 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:52.055761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:52.057044 (+ 1283us) server_negotiation.cc:407] Connection header received
0809 06:46:52.058804 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:52.058808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:52.058875 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:52.058957 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:52.060556 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.061360 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.063664 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.064143 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.064237 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.069928 (+ 5691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:48741: BlockingRecv error: recv got EOF from 127.27.209.195:48741 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:46:52.070989 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:52.056099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:48741)
0809 06:46:52.056861 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:52.056919 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:52.058555 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:52.059158 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:52.059166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:52.059776 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:52.060396 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.060423 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.061551 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.061558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.063481 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.063497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.064387 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.064394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.064587 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.065312 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:52.065360 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:52.069756 (+ 4396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
W20260809 06:46:52.071653 10022 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:36411 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.209.254:36411: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
I20260809 06:46:52.077546 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:52.061773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.27.209.254:36411)
0809 06:46:52.062377 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:52.062381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:52.062394 (+ 13us) server_negotiation.cc:407] Connection header received
0809 06:46:52.062432 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:52.062435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:52.062480 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:52.062557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:52.064359 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.065209 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.066996 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.067508 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.067636 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.070706 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:46:52.070715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:46:52.070732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:46:52.070767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:46:52.074329 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:52.075046 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:52.075054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:52.075061 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:52.075276 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:46:52.075677 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:46:52.075682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:46:52.075685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:46:52.076184 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:46:52.076478 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 06:46:52.076901 (+ 423us) server_negotiation.cc:299] Negotiation successful
0809 06:46:52.077264 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":385,"threads_started":1}
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:52.079771 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:52.072188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:47055)
0809 06:46:52.072415 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:52.072436 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:52.072552 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:52.072853 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:52.072858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:52.073257 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:52.073439 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.073450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.074492 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.074497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.075395 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.075408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.076149 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.076155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.076320 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.076802 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:52.076820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:52.079507 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260809 06:46:52.080564 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:52.072314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:47055 (local address 127.27.209.254:36411)
0809 06:46:52.072464 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:52.072468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:52.072484 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:46:52.072648 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:52.072652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:52.072708 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:52.072784 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:52.073574 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.074347 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.075554 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.076011 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.076125 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.079768 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:47055: BlockingRecv error: recv got EOF from 127.27.209.195:47055 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:52.088071 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:52.080522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:54709)
0809 06:46:52.080768 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:52.080786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:52.080881 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:52.081191 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:52.081196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:52.081500 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:52.081699 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.081712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.082614 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.082619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.083507 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.083518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.084240 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.084246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.084438 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.084999 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:52.085017 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:52.087783 (+ 2766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 06:46:52.088361 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:52.080603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:54709 (local address 127.27.209.254:36411)
0809 06:46:52.080869 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:52.080874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:52.080955 (+ 81us) server_negotiation.cc:407] Connection header received
0809 06:46:52.081006 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:52.081010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:52.081070 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:52.081167 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:52.081825 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.082482 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.083655 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.084096 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.084213 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.087992 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:54709: BlockingRecv error: recv got EOF from 127.27.209.195:54709 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260809 06:46:52.088375 10022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:52.196442 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:52.188159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:55285)
0809 06:46:52.188687 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:52.188706 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:52.188882 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:52.189202 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:52.189206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:52.189759 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:52.189957 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.189968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.190952 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.190958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.191942 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.191956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.192673 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.192679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.192841 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.193384 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:52.193401 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:52.196147 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":305,"threads_started":1}
W20260809 06:46:52.196602 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:52.188185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:55285 (local address 127.27.209.254:36411)
0809 06:46:52.188440 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:52.188447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:52.188796 (+ 349us) server_negotiation.cc:407] Connection header received
0809 06:46:52.189007 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:52.189011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:52.189073 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:52.189151 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:52.190094 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.190806 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.192075 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.192501 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.192609 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.196379 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:55285: BlockingRecv error: recv got EOF from 127.27.209.193:55285 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:52 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:52.648701 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:52.640526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:41101)
0809 06:46:52.640992 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:52.641011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:52.641154 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:52.641488 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:52.641492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:52.642093 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:52.642299 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.642310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.643336 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.643342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.644313 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:52.644325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.645039 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.645045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:52.645208 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.645726 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:52.645742 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:52.648395 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":259,"threads_started":1}
W20260809 06:46:52.648918 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:52.640585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:41101 (local address 127.27.209.254:36411)
0809 06:46:52.640779 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:52.640785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:52.641096 (+ 311us) server_negotiation.cc:407] Connection header received
0809 06:46:52.641283 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:52.641287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:52.641350 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:52.641433 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:52.642439 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.643197 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.644449 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:52.644875 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:52.644982 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:52.648594 (+ 3612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:41101: BlockingRecv error: recv got EOF from 127.27.209.194:41101 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 06:46:53 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:53 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:53.101348 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:53.088989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:47691)
0809 06:46:53.089516 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:53.089531 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:53.089676 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:53.090035 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:53.090041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:53.090528 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:53.090724 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:53.090734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.091808 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.091814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:53.092781 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:53.092794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.093564 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.093571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:53.093725 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:53.098109 (+ 4384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:53.098133 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:53.101093 (+ 2960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":292,"threads_started":1}
W20260809 06:46:53.101572 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:53.089062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:47691 (local address 127.27.209.254:36411)
0809 06:46:53.089267 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:53.089273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:53.089618 (+ 345us) server_negotiation.cc:407] Connection header received
0809 06:46:53.089793 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:53.089797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:53.089876 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:53.089957 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:53.090875 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.091668 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.092923 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.093384 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.093495 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:53.101318 (+ 7823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:47691: BlockingRecv error: recv got EOF from 127.27.209.195:47691 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 06:46:53 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:53 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:53.205174 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:53.197256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:48053)
0809 06:46:53.197745 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:53.197760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:53.197878 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:53.198147 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:53.198151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:53.198569 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:53.198760 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:53.198769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.199806 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.199810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:53.200787 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:53.200800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.201433 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.201438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:53.201547 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:53.202093 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:53.202110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:53.204853 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
W20260809 06:46:53.205291 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:53.197327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:48053 (local address 127.27.209.254:36411)
0809 06:46:53.197559 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:53.197565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:53.197838 (+ 273us) server_negotiation.cc:407] Connection header received
0809 06:46:53.197969 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:53.197973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:53.198037 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:53.198117 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:53.198889 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.199689 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.200908 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.201306 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.201395 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:53.205060 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:48053: BlockingRecv error: recv got EOF from 127.27.209.193:48053 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 06:46:53 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:53 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:53.657361 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:53.649545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:60171)
0809 06:46:53.650043 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:53.650059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:53.650170 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:53.650427 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:53.650430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:53.650854 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:53.651029 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:53.651037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.652004 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.652008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:53.652907 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:53.652919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.653543 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.653547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:53.653658 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:53.654184 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:53.654201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:53.657043 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
W20260809 06:46:53.657509 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:53.649627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:60171 (local address 127.27.209.254:36411)
0809 06:46:53.649860 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:53.649866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:53.650132 (+ 266us) server_negotiation.cc:407] Connection header received
0809 06:46:53.650257 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:53.650261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:53.650321 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:53.650396 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:53.651138 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.651897 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.653026 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:53.653430 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:53.653523 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:53.657282 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:60171: BlockingRecv error: recv got EOF from 127.27.209.194:60171 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 06:46:54 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:54 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:54.110539 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:54.102240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:34263)
0809 06:46:54.102735 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:54.102751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:54.102867 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:54.103147 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:54.103150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:54.103573 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:54.103751 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:54.103762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.104797 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.104801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:54.105718 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:54.105729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.106598 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.106604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:54.106745 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:54.107433 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:54.107456 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:54.110281 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":262,"threads_started":1}
W20260809 06:46:54.110709 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:54.102412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:34263 (local address 127.27.209.254:36411)
0809 06:46:54.102635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:54.102642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:54.102821 (+ 179us) server_negotiation.cc:407] Connection header received
0809 06:46:54.102954 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:54.102957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:54.103040 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:54.103120 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:54.103915 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.104673 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.105864 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.106429 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.106549 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:54.110472 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:34263: BlockingRecv error: recv got EOF from 127.27.209.195:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 06:46:54 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:54 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:54.214184 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:54.206169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:46393)
0809 06:46:54.206687 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:54.206703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:54.206829 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:54.207090 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:54.207093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:54.207427 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:54.207648 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:54.207659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.208645 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.208648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:54.209543 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:54.209554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.210247 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.210251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:54.210377 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:54.210907 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:54.210925 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:54.213874 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
W20260809 06:46:54.214447 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:54.206271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:46393 (local address 127.27.209.254:36411)
0809 06:46:54.206435 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:54.206439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:54.206768 (+ 329us) server_negotiation.cc:407] Connection header received
0809 06:46:54.206916 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:54.206919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:54.206980 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:54.207056 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:54.207808 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.208523 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.209690 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.210126 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.210222 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:54.214080 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:46393: BlockingRecv error: recv got EOF from 127.27.209.193:46393 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 06:46:54 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:54 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:54.665778 10035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:54.658218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:50089)
0809 06:46:54.658730 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:54.658746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:54.658859 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:54.659185 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:54.659188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:54.659528 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:54.659791 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:54.659806 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.660751 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.660754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:54.661626 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:54.661636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.662237 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.662241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:54.662366 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:54.662895 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:54.662911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:54.665530 (+ 2619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
W20260809 06:46:54.665913 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:54.658282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:50089 (local address 127.27.209.254:36411)
0809 06:46:54.658493 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:54.658499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:54.658831 (+ 332us) server_negotiation.cc:407] Connection header received
0809 06:46:54.658984 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:54.658988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:54.659071 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:54.659149 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:54.659941 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.660638 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.661726 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:54.662136 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:54.662222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:54.665706 (+ 3484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:50089: BlockingRecv error: recv got EOF from 127.27.209.194:50089 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 06:46:55 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:55 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:55.119374 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:55.111490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:35075)
0809 06:46:55.112072 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:55.112087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:55.112206 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:55.112483 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:55.112486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:55.112830 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:55.113005 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:55.113015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.114124 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.114129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:55.115209 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:55.115223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.115871 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.115874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:55.115988 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:55.116524 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:55.116541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:55.119116 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":286,"threads_started":1}
W20260809 06:46:55.119526 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:55.111604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:35075 (local address 127.27.209.254:36411)
0809 06:46:55.111772 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:55.111776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:55.112152 (+ 376us) server_negotiation.cc:407] Connection header received
0809 06:46:55.112294 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:55.112297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:55.112373 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:55.112453 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:55.113152 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.113982 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.115329 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.115767 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.115861 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:55.119301 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:35075: BlockingRecv error: recv got EOF from 127.27.209.195:35075 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 06:46:55 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:55 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:55.222980 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:55.215076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:39677)
0809 06:46:55.215596 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:55.215613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:55.215727 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:55.215986 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:55.215989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:55.216404 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:55.216561 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:55.216569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.217497 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.217500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:55.218455 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:55.218466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.219245 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.219252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:55.219408 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:55.220109 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:55.220128 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:55.222753 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":283,"threads_started":1}
W20260809 06:46:55.223165 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:55.215147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:39677 (local address 127.27.209.254:36411)
0809 06:46:55.215306 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:55.215310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:55.215677 (+ 367us) server_negotiation.cc:407] Connection header received
0809 06:46:55.215814 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:55.215818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:55.215877 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:55.215954 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:55.216664 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.217398 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.218593 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.219091 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.219193 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:55.222951 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:39677: BlockingRecv error: recv got EOF from 127.27.209.193:39677 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 06:46:55 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:55 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:55.675247 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:55.666606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:35117)
0809 06:46:55.667122 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:55.667138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:55.667273 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:55.667616 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:55.667620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:55.668026 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:55.668174 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:55.668183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.669304 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.669310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:55.670641 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:55.670656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.671365 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.671371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:55.671573 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:55.672172 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:55.672189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:55.674942 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":285,"threads_started":1}
W20260809 06:46:55.675410 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:55.666690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:35117 (local address 127.27.209.254:36411)
0809 06:46:55.666913 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:55.666920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:55.667221 (+ 301us) server_negotiation.cc:407] Connection header received
0809 06:46:55.667400 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:55.667405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:55.667468 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:55.667572 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:55.668324 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.669177 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.670778 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:55.671218 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:55.671339 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:55.675149 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:35117: BlockingRecv error: recv got EOF from 127.27.209.194:35117 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:46:56 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:56 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:56.128525 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:56.120336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:33757)
0809 06:46:56.120818 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:56.120834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:56.120970 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:56.121308 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:56.121313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:56.121713 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:56.121877 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:56.121886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.123031 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.123035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:56.124153 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:56.124167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.124842 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.124848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:56.125010 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:56.125526 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:56.125543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:56.128295 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":278,"threads_started":1}
W20260809 06:46:56.128685 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:56.120393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:33757 (local address 127.27.209.254:36411)
0809 06:46:56.120591 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:56.120598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:56.120918 (+ 320us) server_negotiation.cc:407] Connection header received
0809 06:46:56.121084 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:56.121088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:56.121165 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:56.121247 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:56.122012 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.122886 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.124297 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.124697 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.124808 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:56.128480 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:33757: BlockingRecv error: recv got EOF from 127.27.209.195:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 06:46:56 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:56 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:56.232402 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:56.224051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:60637)
0809 06:46:56.224611 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:56.224627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:56.224764 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:56.225086 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:56.225091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:56.225551 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:56.225714 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:56.225723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.226779 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.226784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:56.227722 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:56.227735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.228581 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.228587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:56.228753 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:56.229319 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:56.229339 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:56.232059 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":275,"threads_started":1}
W20260809 06:46:56.232534 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:56.224181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:60637 (local address 127.27.209.254:36411)
0809 06:46:56.224422 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:56.224428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:56.224716 (+ 288us) server_negotiation.cc:407] Connection header received
0809 06:46:56.224884 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:56.224889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:56.224952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:56.225033 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:56.225877 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.226611 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.227881 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.228427 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.228541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:56.232294 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:60637: BlockingRecv error: recv got EOF from 127.27.209.193:60637 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 06:46:56 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:56 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:56.689563 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:56.676173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:44259)
0809 06:46:56.676640 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:56.676656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:56.676766 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:56.677058 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:56.677061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:56.677443 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:56.677641 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:56.677656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.679012 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.679018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:56.680018 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:56.680033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.680730 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.680735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:56.680858 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:56.685777 (+ 4919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:56.685797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:56.689223 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":221,"threads_started":1}
W20260809 06:46:56.692005 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:56.676354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:44259 (local address 127.27.209.254:36411)
0809 06:46:56.676568 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:56.676575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:56.676728 (+ 153us) server_negotiation.cc:407] Connection header received
0809 06:46:56.676871 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:56.676875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:56.676946 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:56.677024 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:56.677793 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.678889 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.680153 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:56.680591 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:56.680683 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:56.691774 (+ 11091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:44259: BlockingRecv error: recv got EOF from 127.27.209.194:44259 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 06:46:57 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:57 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:57.137125 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:57.129405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:34397)
0809 06:46:57.129898 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:57.129914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:57.130022 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:57.130299 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:57.130303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:57.130708 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:57.130880 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:57.130890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.131870 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.131874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:57.132771 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:57.132780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.133411 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.133415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:57.133533 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:57.134110 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:57.134129 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:57.136874 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
W20260809 06:46:57.137388 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:57.129484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:34397 (local address 127.27.209.254:36411)
0809 06:46:57.129707 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:57.129713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:57.129986 (+ 273us) server_negotiation.cc:407] Connection header received
0809 06:46:57.130108 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:57.130112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:57.130190 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:57.130272 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:57.130986 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.131766 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.132873 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.133302 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.133393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:57.137095 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:34397: BlockingRecv error: recv got EOF from 127.27.209.195:34397 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 06:46:57 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:57 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:57.241154 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:57.233264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:57475)
0809 06:46:57.233771 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:57.233788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:57.233905 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:57.234174 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:57.234177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:57.234618 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:57.234790 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:57.234800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.235871 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.235875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:57.236783 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:57.236795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.237420 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.237424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:57.237542 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:57.238121 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:57.238139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:57.240926 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":259,"threads_started":1}
W20260809 06:46:57.241333 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:57.233312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:57475 (local address 127.27.209.254:36411)
0809 06:46:57.233509 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:57.233516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:57.233861 (+ 345us) server_negotiation.cc:407] Connection header received
0809 06:46:57.234007 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:57.234011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:57.234074 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:57.234143 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:57.234898 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.235744 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.236896 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.237307 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.237393 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:57.241137 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:57475: BlockingRecv error: recv got EOF from 127.27.209.193:57475 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 06:46:57 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:57 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:57.702390 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:57.693878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:45987)
0809 06:46:57.694458 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:57.694476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:57.694610 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:57.695032 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:57.695036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:57.695594 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:57.695823 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:57.695833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.696915 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.696919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:57.697823 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:57.697836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.698573 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.698580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:57.698747 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:57.699310 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:57.699328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:57.702180 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":327,"threads_started":1}
W20260809 06:46:57.702585 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:57.693939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:45987 (local address 127.27.209.254:36411)
0809 06:46:57.694177 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:57.694183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:57.694555 (+ 372us) server_negotiation.cc:407] Connection header received
0809 06:46:57.694790 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:57.694796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:57.694876 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:57.694965 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:57.695971 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.696783 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.697964 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:57.698377 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:57.698480 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:57.702366 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:45987: BlockingRecv error: recv got EOF from 127.27.209.194:45987 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 06:46:58 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:58 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:58.151078 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:58.138162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:60739)
0809 06:46:58.138693 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:58.138709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:58.138849 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:58.139191 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:58.139197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:58.139663 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:58.139831 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:58.139842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.140853 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.140859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:58.141897 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:58.141911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.142643 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.142650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:58.142831 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:58.147257 (+ 4426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:58.147278 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:58.150745 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":277,"threads_started":1}
W20260809 06:46:58.151216 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:58.138226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:60739 (local address 127.27.209.254:36411)
0809 06:46:58.138411 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:58.138418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:58.138789 (+ 371us) server_negotiation.cc:407] Connection header received
0809 06:46:58.138971 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:58.138975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:58.139054 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:58.139137 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:58.139972 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.140708 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.142041 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.142488 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.142600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:58.150973 (+ 8373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:60739: BlockingRecv error: recv got EOF from 127.27.209.195:60739 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 06:46:58 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:58 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:58.250607 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:58.242059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:49213)
0809 06:46:58.242678 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:58.242712 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:58.242853 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:58.243164 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:58.243169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:58.243644 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:58.243837 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:58.243848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.244873 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.244878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:58.245854 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:58.245872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.246733 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.246741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:58.246894 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:58.247514 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:58.247559 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:58.250340 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":288,"threads_started":1}
W20260809 06:46:58.250803 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:58.242218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:49213 (local address 127.27.209.254:36411)
0809 06:46:58.242427 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:58.242433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:58.242808 (+ 375us) server_negotiation.cc:407] Connection header received
0809 06:46:58.242971 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:58.242975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:58.243037 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:58.243117 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:58.243987 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.244735 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.246003 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.246576 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.246671 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:58.250529 (+ 3858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:49213: BlockingRecv error: recv got EOF from 127.27.209.193:49213 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 06:46:58 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:58 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:58.711457 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:58.703274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:60539)
0809 06:46:58.703886 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:58.703903 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:58.704019 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:58.704292 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:58.704295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:58.704871 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:58.705086 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:58.705098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.706241 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.706247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:58.707152 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:58.707165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.707841 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.707846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:58.707953 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:58.708485 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:58.708501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:58.711189 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":271,"threads_started":1}
W20260809 06:46:58.711680 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:58.703395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:60539 (local address 127.27.209.254:36411)
0809 06:46:58.703634 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:58.703639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:58.703971 (+ 332us) server_negotiation.cc:407] Connection header received
0809 06:46:58.704111 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:58.704115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:58.704178 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:58.704266 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:58.705202 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.706093 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.707266 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:58.707709 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:58.707808 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:58.711431 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:60539: BlockingRecv error: recv got EOF from 127.27.209.194:60539 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 09 06:46:59 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:59 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:59.160356 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:59.152074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:53247)
0809 06:46:59.152669 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:59.152685 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:59.152813 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:59.153197 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:59.153202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:59.153721 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:59.153898 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:59.153908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.154950 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.154955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:59.155916 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:59.155930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.156635 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.156641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:59.156797 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:59.157345 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:59.157363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:59.160105 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":243,"threads_started":1}
W20260809 06:46:59.160485 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:59.152206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:53247 (local address 127.27.209.254:36411)
0809 06:46:59.152408 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:59.152415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:59.152756 (+ 341us) server_negotiation.cc:407] Connection header received
0809 06:46:59.152937 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:59.152943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:59.153042 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:59.153126 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:59.154038 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.154812 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.156046 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.156482 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.156597 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:59.160274 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:53247: BlockingRecv error: recv got EOF from 127.27.209.195:53247 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 06:46:59 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:59 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:59.259805 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:59.251498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:56167)
0809 06:46:59.252063 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:59.252078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:59.252208 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:59.252518 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:59.252523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:59.252951 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:59.253104 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:59.253114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.254110 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.254116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:59.255033 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:59.255045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.255810 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.255817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:59.255980 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:59.256549 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:59.256566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:59.259481 (+ 2915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
W20260809 06:46:59.260017 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:59.251674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:56167 (local address 127.27.209.254:36411)
0809 06:46:59.251907 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:59.251913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:59.252153 (+ 240us) server_negotiation.cc:407] Connection header received
0809 06:46:59.252319 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:59.252325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:59.252392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:59.252471 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:59.253240 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.253975 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.255165 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.255642 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.255751 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:59.259746 (+ 3995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:56167: BlockingRecv error: recv got EOF from 127.27.209.193:56167 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 06:46:59 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:46:59 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:46:59.720723 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:46:59.712369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:58659)
0809 06:46:59.712888 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 06:46:59.712906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:46:59.713024 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:46:59.713376 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:46:59.713380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:46:59.713799 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:46:59.713982 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:59.713992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.715412 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.715416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:59.716405 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:46:59.716419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.717145 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.717150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:46:59.717268 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:59.717797 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:46:59.717813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:46:59.720483 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
W20260809 06:46:59.720933 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:46:59.712432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:58659 (local address 127.27.209.254:36411)
0809 06:46:59.712594 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 06:46:59.712598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:46:59.712988 (+ 390us) server_negotiation.cc:407] Connection header received
0809 06:46:59.713146 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:46:59.713150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:46:59.713215 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:46:59.713324 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:46:59.714128 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.715275 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.716563 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:46:59.716988 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:46:59.717092 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:46:59.720701 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:58659: BlockingRecv error: recv got EOF from 127.27.209.194:58659 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 06:47:00 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:00 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:00.169225 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:00.161310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:53663)
0809 06:47:00.161807 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:00.161823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:00.161937 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:00.162207 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:00.162210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:00.162630 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:00.162826 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:00.162837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.164153 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.164157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:00.165080 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:00.165091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.165727 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.165730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:00.165846 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:00.166395 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:00.166412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:00.169005 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
W20260809 06:47:00.169391 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:00.161382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:53663 (local address 127.27.209.254:36411)
0809 06:47:00.161567 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:00.161571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:00.161887 (+ 316us) server_negotiation.cc:407] Connection header received
0809 06:47:00.162027 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:00.162031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:00.162096 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:00.162177 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:00.162969 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.164012 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.165189 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.165593 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.165683 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:00.169192 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:53663: BlockingRecv error: recv got EOF from 127.27.209.195:53663 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 06:47:00 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:00 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:00.268357 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:00.260685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:50645)
0809 06:47:00.261152 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:00.261168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:00.261280 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:00.261593 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:00.261597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:00.261999 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:00.262167 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:00.262176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.263110 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.263113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:00.264079 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:00.264092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.264756 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.264760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:00.264890 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:00.265392 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:00.265408 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:00.268090 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":271,"threads_started":1}
W20260809 06:47:00.268584 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:00.260758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:50645 (local address 127.27.209.254:36411)
0809 06:47:00.261001 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:00.261008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:00.261254 (+ 246us) server_negotiation.cc:407] Connection header received
0809 06:47:00.261387 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:00.261390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:00.261466 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:00.261544 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:00.262276 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.262993 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.264222 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.264628 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.264723 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:00.268314 (+ 3591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:50645: BlockingRecv error: recv got EOF from 127.27.209.193:50645 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 06:47:00 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:00 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:00.729882 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:00.721574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:46611)
0809 06:47:00.722068 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:00.722085 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:00.722198 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:00.722468 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:00.722471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:00.722875 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:00.723041 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:00.723050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.724076 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.724080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:00.724993 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:00.725004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.725634 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.725638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:00.725756 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:00.726553 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:00.726577 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:00.729640 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":269,"threads_started":1}
W20260809 06:47:00.730026 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:00.721634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:46611 (local address 127.27.209.254:36411)
0809 06:47:00.721798 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:00.721803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:00.722150 (+ 347us) server_negotiation.cc:407] Connection header received
0809 06:47:00.722283 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:00.722287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:00.722353 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:00.722430 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:00.723162 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.723980 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.725099 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:00.725525 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:00.725613 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:00.729827 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:46611: BlockingRecv error: recv got EOF from 127.27.209.194:46611 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 06:47:01 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:01 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:01.178128 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:01.170075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:37089)
0809 06:47:01.170592 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:01.170617 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:01.170734 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:01.170996 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:01.170999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:01.171399 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:01.171597 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:01.171641 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.172579 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.172584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:01.173538 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:01.173551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.174256 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.174261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:01.174391 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:01.174971 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:01.174989 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:01.177770 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":269,"threads_started":1}
W20260809 06:47:01.178244 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:01.170213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:37089 (local address 127.27.209.254:36411)
0809 06:47:01.170440 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:01.170446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:01.170694 (+ 248us) server_negotiation.cc:407] Connection header received
0809 06:47:01.170827 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:01.170830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:01.170891 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:01.170967 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:01.171741 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.172450 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.173646 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.174095 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.174187 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:01.177989 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:37089: BlockingRecv error: recv got EOF from 127.27.209.195:37089 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 06:47:01 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:01 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:01.277542 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:01.269467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:49521)
0809 06:47:01.269989 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:01.270005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:01.270144 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:01.270453 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:01.270463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:01.270861 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:01.271017 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:01.271027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.272127 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.272133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:01.273054 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:01.273066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.273757 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.273763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:01.273922 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:01.274490 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:01.274508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:01.277321 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":299,"threads_started":1}
W20260809 06:47:01.277714 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:01.269553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:49521 (local address 127.27.209.254:36411)
0809 06:47:01.269801 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:01.269807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:01.270096 (+ 289us) server_negotiation.cc:407] Connection header received
0809 06:47:01.270259 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:01.270265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:01.270329 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:01.270407 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:01.271178 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.271992 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.273197 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.273614 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.273723 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:01.277498 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:49521: BlockingRecv error: recv got EOF from 127.27.209.193:49521 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 09 06:47:01 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:01 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:01.738937 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:01.730907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:42879)
0809 06:47:01.731399 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:01.731421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:01.731599 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:01.731930 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:01.731934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:01.732341 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:01.732486 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:01.732495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.733521 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.733527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:01.734468 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:01.734480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.735166 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.735172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:01.735343 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:01.735931 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:01.735948 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:01.738605 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
W20260809 06:47:01.739065 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:01.730936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:42879 (local address 127.27.209.254:36411)
0809 06:47:01.731149 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:01.731157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:01.731510 (+ 353us) server_negotiation.cc:407] Connection header received
0809 06:47:01.731716 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:01.731722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:01.731788 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:01.731875 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:01.732631 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.733387 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.734603 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:01.735025 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:01.735142 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:01.738825 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:42879: BlockingRecv error: recv got EOF from 127.27.209.194:42879 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 06:47:02 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:02 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:02.187500 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:02.179193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:41281)
0809 06:47:02.179776 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:02.179793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:02.179910 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:02.180177 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:02.180180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:02.180535 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:02.180683 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:02.180691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.181934 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.181940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:02.183123 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:02.183137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.183812 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.183816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:02.183927 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:02.184489 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:02.184506 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:02.187260 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":329,"threads_started":1}
W20260809 06:47:02.187690 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:02.179356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:41281 (local address 127.27.209.254:36411)
0809 06:47:02.179523 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:02.179527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:02.179862 (+ 335us) server_negotiation.cc:407] Connection header received
0809 06:47:02.180002 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:02.180006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:02.180068 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:02.180148 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:02.180829 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.181768 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.183250 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.183678 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.183773 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:02.187450 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:41281: BlockingRecv error: recv got EOF from 127.27.209.195:41281 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 06:47:02 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:02 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:02.286612 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:02.278436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:35033)
0809 06:47:02.278986 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:02.279002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:02.279135 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:02.279480 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:02.279485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:02.279957 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:02.280155 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:02.280165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.281161 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.281166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:02.282071 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:02.282084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.282787 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.282793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:02.282954 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:02.283492 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:02.283510 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:02.286285 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
W20260809 06:47:02.286768 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:02.278584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:35033 (local address 127.27.209.254:36411)
0809 06:47:02.278862 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:02.278868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:02.279083 (+ 215us) server_negotiation.cc:407] Connection header received
0809 06:47:02.279249 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:02.279254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:02.279337 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:02.279421 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:02.280309 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.281028 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.282212 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.282640 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.282759 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:02.286506 (+ 3747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:35033: BlockingRecv error: recv got EOF from 127.27.209.193:35033 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 09 06:47:02 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:02 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:02.747773 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:02.739920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:54413)
0809 06:47:02.740424 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:02.740441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:02.740552 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:02.740810 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:02.740813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:02.741212 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:02.741410 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:02.741419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.742390 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.742394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:02.743288 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:02.743307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.743962 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.743966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:02.744089 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:02.744645 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:02.744661 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:02.747401 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
W20260809 06:47:02.747860 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:02.740022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:54413 (local address 127.27.209.254:36411)
0809 06:47:02.740191 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:02.740195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:02.740509 (+ 314us) server_negotiation.cc:407] Connection header received
0809 06:47:02.740632 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:02.740636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:02.740700 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:02.740778 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:02.741526 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.742268 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.743402 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:02.743853 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:02.743946 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:02.747636 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:54413: BlockingRecv error: recv got EOF from 127.27.209.194:54413 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 06:47:03 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:03 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:03.199169 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:03.188466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:44587)
0809 06:47:03.188965 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:03.188982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:03.189098 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:03.189430 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:03.189433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:03.189867 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:03.190057 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:03.190067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.191104 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.191108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:03.192095 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:03.192108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.192826 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.192830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:03.192952 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:03.195991 (+ 3039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:03.196009 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:03.198828 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
W20260809 06:47:03.199314 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:03.188536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:44587 (local address 127.27.209.254:36411)
0809 06:47:03.188750 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:03.188762 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:03.189060 (+ 298us) server_negotiation.cc:407] Connection header received
0809 06:47:03.189243 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:03.189248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:03.189315 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:03.189395 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:03.190224 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.191000 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.192242 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.192669 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.192773 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:03.199040 (+ 6267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:44587: BlockingRecv error: recv got EOF from 127.27.209.195:44587 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 06:47:03 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:03 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:03.295603 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:03.287562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:46483)
0809 06:47:03.288093 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:03.288109 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:03.288233 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:03.288504 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:03.288507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:03.288922 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:03.289085 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:03.289095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.290035 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.290039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:03.290966 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:03.290979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.291596 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.291600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:03.291744 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:03.292340 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:03.292360 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:03.295233 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":304,"threads_started":1}
W20260809 06:47:03.295719 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:03.287634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:46483 (local address 127.27.209.254:36411)
0809 06:47:03.287843 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:03.287847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:03.288175 (+ 328us) server_negotiation.cc:407] Connection header received
0809 06:47:03.288324 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:03.288327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:03.288388 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:03.288464 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:03.289195 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.289913 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.291074 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.291476 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.291599 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:03.295456 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:46483: BlockingRecv error: recv got EOF from 127.27.209.193:46483 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 06:47:03 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:03 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:03.757031 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:03.748697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:41031)
0809 06:47:03.749184 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:03.749201 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:03.749315 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:03.749568 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:03.749571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:03.749995 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:03.750151 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:03.750160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.751179 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.751184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:03.752272 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:03.752288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.753132 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.753138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:03.753288 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:03.753918 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:03.753935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:03.756687 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":279,"threads_started":1}
W20260809 06:47:03.757186 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:03.748764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:41031 (local address 127.27.209.254:36411)
0809 06:47:03.748978 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:03.748985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:03.749272 (+ 287us) server_negotiation.cc:407] Connection header received
0809 06:47:03.749401 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:03.749404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:03.749465 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:03.749540 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:03.750267 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.751044 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.752407 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:03.752984 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:03.753077 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:03.756898 (+ 3821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:41031: BlockingRecv error: recv got EOF from 127.27.209.194:41031 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 09 06:47:04 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:04 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:04.208253 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:04.200103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:53273)
0809 06:47:04.200670 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:04.200691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:04.200834 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:04.201188 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:04.201194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:04.201616 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:04.201779 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:04.201788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.202875 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.202879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:04.203830 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:04.203843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.204525 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.204529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:04.204654 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:04.205202 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:04.205219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:04.208028 (+ 2809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":346,"threads_started":1}
W20260809 06:47:04.208439 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:04.200157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:53273 (local address 127.27.209.254:36411)
0809 06:47:04.200379 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:04.200385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:04.200775 (+ 390us) server_negotiation.cc:407] Connection header received
0809 06:47:04.200940 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:04.200947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:04.201026 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:04.201107 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:04.201916 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.202728 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.203964 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.204414 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.204508 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:04.208216 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:53273: BlockingRecv error: recv got EOF from 127.27.209.195:53273 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 06:47:04 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:04 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:04.304654 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:04.296628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:33213)
0809 06:47:04.297170 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:04.297185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:04.297314 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:04.297615 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:04.297620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:04.298023 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:04.298190 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:04.298199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.299221 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.299225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:04.300219 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:04.300234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.300990 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.300996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:04.301143 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:04.301706 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:04.301724 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:04.304336 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":286,"threads_started":1}
W20260809 06:47:04.304778 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:04.296714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:33213 (local address 127.27.209.254:36411)
0809 06:47:04.296923 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:04.296929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:04.297262 (+ 333us) server_negotiation.cc:407] Connection header received
0809 06:47:04.297423 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:04.297428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:04.297490 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:04.297568 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:04.298336 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.299060 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.300362 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.300816 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.300925 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:04.304549 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:33213: BlockingRecv error: recv got EOF from 127.27.209.193:33213 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 06:47:04 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:04 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:04.766166 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:04.757922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:48919)
0809 06:47:04.758441 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:04.758456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:04.758581 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:04.758896 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:04.758901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:04.759303 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:04.759463 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:04.759472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.760517 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.760522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:04.761441 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:04.761453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.762192 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.762200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:04.762371 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:04.762932 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:04.762948 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:04.765936 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":266,"threads_started":1}
W20260809 06:47:04.766386 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:04.758097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:48919 (local address 127.27.209.254:36411)
0809 06:47:04.758300 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:04.758307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:04.758530 (+ 223us) server_negotiation.cc:407] Connection header received
0809 06:47:04.758702 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:04.758706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:04.758765 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:04.758840 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:04.759620 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.760381 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.761579 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:04.762016 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:04.762125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:04.766164 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:48919: BlockingRecv error: recv got EOF from 127.27.209.194:48919 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 06:47:05 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:05 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:05.217643 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:05.209084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:39211)
0809 06:47:05.209668 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:05.209684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:05.209819 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:05.210169 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:05.210175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:05.210578 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:05.210741 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:05.210751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.211979 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.211984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:05.213109 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:05.213132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.213861 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.213867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:05.214033 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:05.214594 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:05.214611 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:05.217398 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":284,"threads_started":1}
W20260809 06:47:05.217833 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:05.209239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:39211 (local address 127.27.209.254:36411)
0809 06:47:05.209460 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:05.209467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:05.209765 (+ 298us) server_negotiation.cc:407] Connection header received
0809 06:47:05.209935 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:05.209940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:05.210004 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:05.210084 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:05.210880 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.211845 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.213257 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.213709 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.213839 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:05.217581 (+ 3742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:39211: BlockingRecv error: recv got EOF from 127.27.209.195:39211 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 06:47:05 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:05 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:05.313665 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:05.305548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:41399)
0809 06:47:05.306004 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:05.306021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:05.306137 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:05.306432 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:05.306437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:05.306899 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:05.307119 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:05.307135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.308424 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.308428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:05.309320 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:05.309331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.309938 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.309942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:05.310084 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:05.310622 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:05.310638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:05.313424 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
W20260809 06:47:05.313818 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:05.305640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:41399 (local address 127.27.209.254:36411)
0809 06:47:05.305897 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:05.305901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:05.306084 (+ 183us) server_negotiation.cc:407] Connection header received
0809 06:47:05.306219 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:05.306222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:05.306284 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:05.306360 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:05.307277 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.308299 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.309426 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.309833 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.309924 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:05.313612 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:41399: BlockingRecv error: recv got EOF from 127.27.209.193:41399 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 09 06:47:05 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:05 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:05.775311 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:05.767083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:36101)
0809 06:47:05.767524 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:05.767559 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:05.767711 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:05.767984 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:05.767987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:05.768355 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:05.768539 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:05.768549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.769535 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.769539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:05.770474 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:05.770486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.771160 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.771165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:05.771298 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:05.771946 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:05.771964 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:05.774976 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
W20260809 06:47:05.775442 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:05.767228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:36101 (local address 127.27.209.254:36411)
0809 06:47:05.767448 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:05.767454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:05.767660 (+ 206us) server_negotiation.cc:407] Connection header received
0809 06:47:05.767812 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:05.767816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:05.767879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:05.767955 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:05.768684 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.769411 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.770584 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:05.771012 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:05.771099 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:05.775213 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:36101: BlockingRecv error: recv got EOF from 127.27.209.194:36101 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 09 06:47:06 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:06 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.195@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:06.226752 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:06.218658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.195:50755)
0809 06:47:06.219181 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:06.219200 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:06.219344 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:06.219740 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:06.219745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:06.220131 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:06.220311 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:06.220321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.221305 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.221311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:06.222212 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:06.222223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.222945 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.222951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:06.223117 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:06.223719 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:06.223738 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:06.226468 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
W20260809 06:47:06.226879 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:06.218720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:50755 (local address 127.27.209.254:36411)
0809 06:47:06.218904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:06.218910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:06.219273 (+ 363us) server_negotiation.cc:407] Connection header received
0809 06:47:06.219469 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:06.219474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:06.219600 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:06.219697 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:06.220455 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.221170 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.222354 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.222807 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.222915 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:06.226684 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.195:50755: BlockingRecv error: recv got EOF from 127.27.209.195:50755 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 06:47:06 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:06 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.193@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:06.322683 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:06.314850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.193:32921)
0809 06:47:06.315376 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:06.315396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:06.315558 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:06.315865 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:06.315870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:06.316315 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:06.316490 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:06.316499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.317457 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.317463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:06.318350 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:06.318363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.319005 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.319011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:06.319172 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:06.319726 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:06.319743 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:06.322424 (+ 2681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":428,"threads_started":1}
W20260809 06:47:06.322870 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:06.314914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:32921 (local address 127.27.209.254:36411)
0809 06:47:06.315152 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:06.315158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:06.315459 (+ 301us) server_negotiation.cc:407] Connection header received
0809 06:47:06.315666 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:06.315672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:06.315735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:06.315838 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:06.316627 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.317320 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.318482 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.318873 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.318980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:06.322617 (+ 3637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.193:32921: BlockingRecv error: recv got EOF from 127.27.209.193:32921 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 06:47:06 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
Aug 09 06:47:06 dist-test-slave-jpj1 krb5kdc[8396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.209.194@KRBTEST.COM for oryx/127.27.209.254@KRBTEST.COM, Server not found in Kerberos database
I20260809 06:47:06.784281 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:06.776418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36411 (local address 127.27.209.194:43037)
0809 06:47:06.776912 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:06.776930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:06.777042 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:06.777327 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:06.777329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:06.777724 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:06.777892 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:06.777900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.778883 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.778887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:06.779859 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:06.779873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.780567 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.780573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:06.780734 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:06.781319 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:06.781336 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:06.784030 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.209.254:36411: Server oryx/127.27.209.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
W20260809 06:47:06.784406 10027 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:06.776496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:43037 (local address 127.27.209.254:36411)
0809 06:47:06.776697 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:06.776702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:06.777029 (+ 327us) server_negotiation.cc:407] Connection header received
0809 06:47:06.777134 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:06.777138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:06.777204 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:06.777288 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:06.778001 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.778775 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.779964 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:06.780411 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:06.780525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:06.784207 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.209.194:43037: BlockingRecv error: recv got EOF from 127.27.209.194:43037 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:47:07.063012 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9359
I20260809 06:47:07.074862 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:07.064036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.27.209.193:33293)
0809 06:47:07.064610 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:07.064617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:07.064644 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:47:07.064703 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:07.064710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:07.064932 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:07.065079 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:07.066474 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.067521 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.068881 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.069485 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.069607 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:07.072069 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:07.072077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:07.072092 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:07.072128 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:07.074376 (+ 2248us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 06:47:07.074628 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36728: Request ticket server kudu/127.27.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":289,"threads_started":1}
W20260809 06:47:07.074958 10088 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36728: Request ticket server kudu/127.27.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 06:47:07.075042 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9359: Not authorized: Client connection negotiation failed: client connection to 127.27.209.193:33293: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.209.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 06:47:07.075265 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 9359
I20260809 06:47:07.098158 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9608
I20260809 06:47:07.110183 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:07.099145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.27.209.194:39293)
0809 06:47:07.099736 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:07.099743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:07.099776 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:47:07.099855 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:07.099861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:07.100101 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:07.100251 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:07.101593 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.102649 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.104030 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.104581 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.104682 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:07.107123 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:07.107132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:07.107146 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:07.107177 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:07.109633 (+ 2456us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 06:47:07.109897 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54034: Request ticket server kudu/127.27.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":249,"threads_started":1}
W20260809 06:47:07.110303 10092 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54034: Request ticket server kudu/127.27.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 06:47:07.110446 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9608: Not authorized: Client connection negotiation failed: client connection to 127.27.209.194:39293: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.209.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 06:47:07.110597 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 9608
I20260809 06:47:07.136061 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9816
I20260809 06:47:07.147975 10093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:07.137119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.27.209.195:33607)
0809 06:47:07.137685 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:07.137692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:07.137720 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:47:07.137783 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:07.137790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:07.138010 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:07.138179 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:07.139432 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.140497 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.141680 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.142241 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.142338 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:07.144883 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:07.144892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:07.144908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:07.144946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:07.147418 (+ 2472us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 06:47:07.147706 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41808: Request ticket server kudu/127.27.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":263,"threads_started":1}
W20260809 06:47:07.148082 10093 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41808: Request ticket server kudu/127.27.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 06:47:07.148070 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9816: Not authorized: Client connection negotiation failed: client connection to 127.27.209.195:33607: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.209.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 06:47:07.148257 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 9816
I20260809 06:47:07.169540 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 9260
I20260809 06:47:07.170939 9348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:07.286926 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 9260
2026-08-09T06:47:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21236 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T06:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:47:07Z Disabled control of system clock
I20260809 06:47:07.373930 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35139
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:35139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 06:47:07.703878 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:07.704154 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:07.704273 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:07.711333 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:47:07.711431 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:07.711494 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:07.711611 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:47:07.711668 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:47:07.720623 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:35139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:35139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:07.721872 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:07.723656 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:07.734941 10113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:07.735009 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:07.734972 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:07.735575 10104 server_base.cc:1034] running on GCE node
I20260809 06:47:07.736364 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:07.737654 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:07.738859 10104 hybrid_clock.cc:648] HybridClock initialized: now 1786258027738819 us; error 29 us; skew 500 ppm
I20260809 06:47:07.739187 10104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:07.741400 10104 webserver.cc:460] Webserver started at http://127.27.209.254:46811/ using document root <none> and password file <none>
I20260809 06:47:07.742059 10104 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:07.742156 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:07.742430 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:07.745296 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "2a18437ec10d407fa3be1d232848ea52"
format_stamp: "Formatted at 2026-08-09 06:47:07 on dist-test-slave-jpj1"
server_key: "3d15cd9a17f42a0834404a17c1b1fe4f"
server_key_iv: "4fbab6677cce04373ecf72b07478e856"
server_key_version: "encryptionkey@0"
I20260809 06:47:07.746026 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "2a18437ec10d407fa3be1d232848ea52"
format_stamp: "Formatted at 2026-08-09 06:47:07 on dist-test-slave-jpj1"
server_key: "3d15cd9a17f42a0834404a17c1b1fe4f"
server_key_iv: "4fbab6677cce04373ecf72b07478e856"
server_key_version: "encryptionkey@0"
I20260809 06:47:07.751050 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:47:07.754434 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:07.755740 10104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:47:07.755879 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "2a18437ec10d407fa3be1d232848ea52"
format_stamp: "Formatted at 2026-08-09 06:47:07 on dist-test-slave-jpj1"
server_key: "3d15cd9a17f42a0834404a17c1b1fe4f"
server_key_iv: "4fbab6677cce04373ecf72b07478e856"
server_key_version: "encryptionkey@0"
I20260809 06:47:07.756053 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:07.777613 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:07.778412 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:07.778647 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:07.797650 10104 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:35139
I20260809 06:47:07.797688 10210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:35139 every 8 connection(s)
I20260809 06:47:07.798971 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:47:07.802815 10212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:47:07.805553 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 10104
I20260809 06:47:07.805716 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:47:07.806358 28487 external_mini_cluster.cc:1428] Setting key 173fe7b03dde00221e6a603deb9bd465
I20260809 06:47:07.814401 10212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52: Bootstrap starting.
I20260809 06:47:07.817412 10212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52: Neither blocks nor log segments found. Creating new log.
I20260809 06:47:07.818476 10212 log.cc:824] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52: Log is configured to *not* fsync() on all Append() calls
I20260809 06:47:07.819474 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:07.808408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.27.209.254:35139)
0809 06:47:07.809358 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:07.809369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:07.809410 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:47:07.811080 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:07.811105 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:07.811626 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:07.812005 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:07.812611 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.813834 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.815197 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:07.815948 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:07.816122 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:07.817215 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:07.817223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:07.817236 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:07.817806 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:07.817902 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:07.817923 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:07.818022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:07.818246 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 06:47:07.818535 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":315,"threads_started":1}
I20260809 06:47:07.821198 10212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52: No bootstrap required, opened a new log
I20260809 06:47:07.828605 10212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a18437ec10d407fa3be1d232848ea52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35139 } }
I20260809 06:47:07.828943 10212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:47:07.829035 10212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a18437ec10d407fa3be1d232848ea52, State: Initialized, Role: FOLLOWER
I20260809 06:47:07.829605 10212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a18437ec10d407fa3be1d232848ea52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35139 } }
I20260809 06:47:07.829840 10212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:47:07.829959 10212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:47:07.830106 10212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:47:07.832473 10212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a18437ec10d407fa3be1d232848ea52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35139 } }
I20260809 06:47:07.832935 10212 leader_election.cc:302] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a18437ec10d407fa3be1d232848ea52; no voters:
I20260809 06:47:07.833467 10212 leader_election.cc:288] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:47:07.833595 10217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:47:07.834249 10217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [term 1 LEADER]: Becoming Leader. State: Replica: 2a18437ec10d407fa3be1d232848ea52, State: Running, Role: LEADER
I20260809 06:47:07.834838 10217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a18437ec10d407fa3be1d232848ea52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35139 } }
I20260809 06:47:07.835414 10212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:47:07.837674 10220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a18437ec10d407fa3be1d232848ea52. Latest consensus state: current_term: 1 leader_uuid: "2a18437ec10d407fa3be1d232848ea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a18437ec10d407fa3be1d232848ea52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35139 } } }
I20260809 06:47:07.837786 10219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a18437ec10d407fa3be1d232848ea52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a18437ec10d407fa3be1d232848ea52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 35139 } } }
I20260809 06:47:07.838043 10220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [sys.catalog]: This master's current role is: LEADER
I20260809 06:47:07.838135 10219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52 [sys.catalog]: This master's current role is: LEADER
I20260809 06:47:07.838902 10225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:47:07.844534 10225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:47:07.853952 10225 catalog_manager.cc:1347] Generated new cluster ID: 576382ba365d41dc95e86c0919a17d69
I20260809 06:47:07.854089 10225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:47:07.883342 10225 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:47:07.885185 10225 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:47:07.895557 10225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a18437ec10d407fa3be1d232848ea52: Generated new TSK 0
I20260809 06:47:07.896554 10225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:47:07.904359 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35139
--builtin_ntp_servers=127.27.209.212:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 06:47:08.233467 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:08.233757 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:08.233867 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:08.241230 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:08.241339 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:08.241482 10242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:47:08.250381 10242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:08.251711 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:08.253505 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:08.266619 10250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:08.266590 10255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:08.266718 10252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:08.267360 10242 server_base.cc:1034] running on GCE node
I20260809 06:47:08.268002 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:08.268893 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:08.270095 10242 hybrid_clock.cc:648] HybridClock initialized: now 1786258028270031 us; error 58 us; skew 500 ppm
I20260809 06:47:08.270404 10242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:08.275750 10242 webserver.cc:460] Webserver started at http://127.27.209.193:39429/ using document root <none> and password file <none>
I20260809 06:47:08.276481 10242 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:08.276588 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:08.276909 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:08.279815 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "decb85d542bf425a8d33c9e47b24d079"
format_stamp: "Formatted at 2026-08-09 06:47:08 on dist-test-slave-jpj1"
server_key: "fa7eea3b6b39e8ef440b0f3547694154"
server_key_iv: "72448c61496fe062f2f578de6ab9184a"
server_key_version: "encryptionkey@0"
I20260809 06:47:08.280615 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "decb85d542bf425a8d33c9e47b24d079"
format_stamp: "Formatted at 2026-08-09 06:47:08 on dist-test-slave-jpj1"
server_key: "fa7eea3b6b39e8ef440b0f3547694154"
server_key_iv: "72448c61496fe062f2f578de6ab9184a"
server_key_version: "encryptionkey@0"
I20260809 06:47:08.285881 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:47:08.289288 10263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:08.290613 10242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:47:08.290777 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "decb85d542bf425a8d33c9e47b24d079"
format_stamp: "Formatted at 2026-08-09 06:47:08 on dist-test-slave-jpj1"
server_key: "fa7eea3b6b39e8ef440b0f3547694154"
server_key_iv: "72448c61496fe062f2f578de6ab9184a"
server_key_version: "encryptionkey@0"
I20260809 06:47:08.290977 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:08.326383 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:08.327296 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:08.327628 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:08.328610 10242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:08.330281 10242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:08.330386 10242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:08.330514 10242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:08.330600 10242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:08.362212 10242 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:45581
I20260809 06:47:08.362286 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:45581 every 8 connection(s)
I20260809 06:47:08.363502 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:47:08.370986 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 10242
I20260809 06:47:08.371151 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:47:08.371852 28487 external_mini_cluster.cc:1428] Setting key d054c0114113c2c56e21251f6d436b7e
I20260809 06:47:08.374797 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35139
--builtin_ntp_servers=127.27.209.212:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 06:47:08.376055 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:08.366006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:44415 (local address 127.27.209.254:35139)
0809 06:47:08.366434 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:08.366438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:08.367707 (+ 1269us) server_negotiation.cc:407] Connection header received
0809 06:47:08.369504 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:08.369509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:08.369604 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:08.369704 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:08.370810 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.371608 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.373256 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.373768 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.373863 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:08.375330 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:08.375334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:08.375337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:08.375354 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:08.375397 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:08.375409 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:08.375477 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:08.375731 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 06:47:08.375805 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":204,"threads_started":1}
I20260809 06:47:08.377375 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:08.366633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35139 (local address 127.27.209.193:44415)
0809 06:47:08.367514 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:08.367596 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:08.369229 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:08.369947 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:08.369956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:08.370044 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:08.370638 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:08.370656 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.371750 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.371753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:08.373135 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:08.373150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.373988 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.373996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:08.374284 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:08.375002 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:47:08.375006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:08.375061 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:47:08.375160 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:08.375496 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:08.375506 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:08.375513 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:47:08.375724 (+ 211us) client_negotiation.cc:240] Negotiation successful
0809 06:47:08.376110 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
I20260809 06:47:08.378750 10431 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35139
I20260809 06:47:08.379043 10431 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:08.379813 10431 heartbeater.cc:511] Master 127.27.209.254:35139 requested a full tablet report, sending...
I20260809 06:47:08.381755 10149 ts_manager.cc:194] Registered new tserver with Master: decb85d542bf425a8d33c9e47b24d079 (127.27.209.193:45581)
I20260809 06:47:08.383250 10149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.193:44415
W20260809 06:47:08.704089 10436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:08.704365 10436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:08.704458 10436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:08.711596 10436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:08.711705 10436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:08.711815 10436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:47:08.720621 10436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:08.721856 10436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:08.723613 10436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:08.736344 10446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:08.736418 10443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:08.736392 10444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:08.737373 10436 server_base.cc:1034] running on GCE node
I20260809 06:47:08.738010 10436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:08.738942 10436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:08.740151 10436 hybrid_clock.cc:648] HybridClock initialized: now 1786258028740091 us; error 50 us; skew 500 ppm
I20260809 06:47:08.740480 10436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:08.742774 10436 webserver.cc:460] Webserver started at http://127.27.209.194:34949/ using document root <none> and password file <none>
I20260809 06:47:08.743477 10436 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:08.743628 10436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:08.743956 10436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:08.746776 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "f91a2371d492404ab1f9de9c578b8b54"
format_stamp: "Formatted at 2026-08-09 06:47:08 on dist-test-slave-jpj1"
server_key: "30751d1f43556a115658266073d25375"
server_key_iv: "1ed65f29fd899b7eb68449ab0bbf847d"
server_key_version: "encryptionkey@0"
I20260809 06:47:08.747587 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "f91a2371d492404ab1f9de9c578b8b54"
format_stamp: "Formatted at 2026-08-09 06:47:08 on dist-test-slave-jpj1"
server_key: "30751d1f43556a115658266073d25375"
server_key_iv: "1ed65f29fd899b7eb68449ab0bbf847d"
server_key_version: "encryptionkey@0"
I20260809 06:47:08.752775 10436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:47:08.756170 10454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:08.757411 10436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:47:08.757566 10436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "f91a2371d492404ab1f9de9c578b8b54"
format_stamp: "Formatted at 2026-08-09 06:47:08 on dist-test-slave-jpj1"
server_key: "30751d1f43556a115658266073d25375"
server_key_iv: "1ed65f29fd899b7eb68449ab0bbf847d"
server_key_version: "encryptionkey@0"
I20260809 06:47:08.757782 10436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:08.776644 10436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:08.777455 10436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:08.777743 10436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:08.778702 10436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:08.780155 10436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:08.780272 10436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:08.780366 10436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:08.780422 10436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:08.810827 10436 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:38827
I20260809 06:47:08.810856 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:38827 every 8 connection(s)
I20260809 06:47:08.812419 10436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:47:08.821121 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 10436
I20260809 06:47:08.821295 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:47:08.821853 28487 external_mini_cluster.cc:1428] Setting key 1a5f3735697f403b7c720c4a59f8795f
I20260809 06:47:08.824762 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35139
--builtin_ntp_servers=127.27.209.212:44337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 06:47:08.824937 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:08.815043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:51515 (local address 127.27.209.254:35139)
0809 06:47:08.815244 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:08.815250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:08.816747 (+ 1497us) server_negotiation.cc:407] Connection header received
0809 06:47:08.818433 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:08.818437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:08.818501 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:08.818594 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:08.819615 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.820333 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.822364 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.822755 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.822866 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:08.824151 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:08.824157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:08.824161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:08.824181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:08.824219 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:08.824235 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:08.824330 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:08.824538 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 06:47:08.824644 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:47:08.825763 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:08.815528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35139 (local address 127.27.209.194:51515)
0809 06:47:08.816566 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:08.816619 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:08.818189 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:08.818809 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:08.818817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:08.818896 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:08.819463 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:08.819479 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.820445 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.820449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:08.822227 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:08.822249 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:08.822947 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:08.822952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:08.823146 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:08.823862 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:47:08.823864 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:08.823901 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:47:08.823973 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:08.824334 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:08.824343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:08.824349 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 06:47:08.824483 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 06:47:08.824764 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":303,"threads_started":1}
I20260809 06:47:08.826959 10636 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35139
I20260809 06:47:08.827217 10636 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:08.827975 10636 heartbeater.cc:511] Master 127.27.209.254:35139 requested a full tablet report, sending...
I20260809 06:47:08.829279 10149 ts_manager.cc:194] Registered new tserver with Master: f91a2371d492404ab1f9de9c578b8b54 (127.27.209.194:38827)
I20260809 06:47:08.830056 10149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.194:51515
W20260809 06:47:09.152312 10639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:09.152580 10639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:09.152707 10639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:09.159732 10639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:09.159826 10639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:09.159966 10639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:47:09.168763 10639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:35139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:09.169961 10639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:09.171712 10639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:09.184226 10645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:09.184299 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:09.184266 10646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:09.185124 10639 server_base.cc:1034] running on GCE node
I20260809 06:47:09.185756 10639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:09.186685 10639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:09.187881 10639 hybrid_clock.cc:648] HybridClock initialized: now 1786258029187815 us; error 54 us; skew 500 ppm
I20260809 06:47:09.188179 10639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:09.190289 10639 webserver.cc:460] Webserver started at http://127.27.209.195:42713/ using document root <none> and password file <none>
I20260809 06:47:09.190974 10639 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:09.191073 10639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:09.191391 10639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:09.194233 10639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "cc88f89726094a27adcb9fd100aed136"
format_stamp: "Formatted at 2026-08-09 06:47:09 on dist-test-slave-jpj1"
server_key: "76ca3d0afd9dbdfacc11db87bb2c60af"
server_key_iv: "919f368a5d53115eb2a31ebd55e1688f"
server_key_version: "encryptionkey@0"
I20260809 06:47:09.195039 10639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "cc88f89726094a27adcb9fd100aed136"
format_stamp: "Formatted at 2026-08-09 06:47:09 on dist-test-slave-jpj1"
server_key: "76ca3d0afd9dbdfacc11db87bb2c60af"
server_key_iv: "919f368a5d53115eb2a31ebd55e1688f"
server_key_version: "encryptionkey@0"
I20260809 06:47:09.200049 10639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 06:47:09.203354 10658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:09.204620 10639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:47:09.204787 10639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "cc88f89726094a27adcb9fd100aed136"
format_stamp: "Formatted at 2026-08-09 06:47:09 on dist-test-slave-jpj1"
server_key: "76ca3d0afd9dbdfacc11db87bb2c60af"
server_key_iv: "919f368a5d53115eb2a31ebd55e1688f"
server_key_version: "encryptionkey@0"
I20260809 06:47:09.204977 10639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:09.232851 10639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:09.233704 10639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:09.233990 10639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:09.234987 10639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:09.236429 10639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:09.236524 10639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:09.236629 10639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:09.236712 10639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:09.267304 10639 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:34329
I20260809 06:47:09.267361 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:34329 every 8 connection(s)
I20260809 06:47:09.268566 10639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:47:09.269760 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 10639
I20260809 06:47:09.269945 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:47:09.270536 28487 external_mini_cluster.cc:1428] Setting key 5ce01720d7b797d0e63bf1ad91064a85
I20260809 06:47:09.281055 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:09.271315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:38033 (local address 127.27.209.254:35139)
0809 06:47:09.271524 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:09.271552 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:09.273199 (+ 1647us) server_negotiation.cc:407] Connection header received
0809 06:47:09.274945 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:09.274951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:09.275032 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:09.275116 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:09.276232 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.276962 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.278459 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.278863 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.278972 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:09.280271 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:09.280275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:09.280278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:09.280295 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:09.280326 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:09.280337 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:09.280411 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:09.280627 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 06:47:09.280723 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:47:09.281968 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:09.271976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:35139 (local address 127.27.209.195:38033)
0809 06:47:09.273047 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:09.273100 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:09.274693 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:09.275359 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:09.275367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:09.275459 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:09.276069 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:09.276087 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.277115 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.277119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:09.278319 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:09.278332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.279074 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.279081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:09.279297 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:09.279976 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:47:09.279979 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:09.280021 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:47:09.280093 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:09.280422 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:09.280434 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:09.280441 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:47:09.280588 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 06:47:09.280974 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":319,"threads_started":1}
I20260809 06:47:09.283201 10836 heartbeater.cc:347] Connected to a master server at 127.27.209.254:35139
I20260809 06:47:09.283485 10836 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:09.284385 10836 heartbeater.cc:511] Master 127.27.209.254:35139 requested a full tablet report, sending...
I20260809 06:47:09.285832 10148 ts_manager.cc:194] Registered new tserver with Master: cc88f89726094a27adcb9fd100aed136 (127.27.209.195:34329)
I20260809 06:47:09.286659 10148 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.195:38033
I20260809 06:47:09.299273 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 06:47:09.305197 10434 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:09.303721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.27.209.254:35139)
0809 06:47:09.303908 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:09.303912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:09.304347 (+ 435us) server_negotiation.cc:407] Connection header received
0809 06:47:09.304477 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:09.304481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:09.304548 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:09.304627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:09.304964 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34768: BlockingRecv error: recv got EOF from 127.0.0.1:34768 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:47:09.305965 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 10242
I20260809 06:47:09.313736 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:09.306944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.27.209.193:45581)
0809 06:47:09.307443 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:09.307450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:09.307475 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:47:09.307527 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:09.307563 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:09.307789 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:09.307911 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:09.308269 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.309284 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.310608 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.311158 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.311269 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:09.312322 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:09.312330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:09.312342 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:09.312930 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:09.312976 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:09.312995 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:09.313082 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:09.313220 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 06:47:09.313326 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":253,"threads_started":1}
I20260809 06:47:09.314963 10426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:09.395764 10431 heartbeater.cc:503] Master 127.27.209.254:35139 was elected leader, sending a full tablet report...
I20260809 06:47:09.397011 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 10242
W20260809 06:47:09.419868 10133 connection.cc:535] server connection from 127.27.209.193:44415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 06:47:09.420202 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 10436
I20260809 06:47:09.427706 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:09.421123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.27.209.194:38827)
0809 06:47:09.421709 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:09.421718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:09.421748 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:47:09.421812 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:09.421819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:09.422025 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:09.422156 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:09.422490 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.423486 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.424693 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.425236 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.425328 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:09.426215 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:09.426224 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:09.426237 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:09.426819 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:09.426867 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:09.426886 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:09.426972 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:09.427065 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 06:47:09.427219 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":309,"threads_started":1}
I20260809 06:47:09.428949 10629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:09.509327 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 10436
I20260809 06:47:09.533129 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 10639
I20260809 06:47:09.540937 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:09.534101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.27.209.195:34329)
0809 06:47:09.534663 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:09.534670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:09.534701 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:47:09.534765 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:09.534772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:09.534986 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:09.535109 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:09.535518 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.536651 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.537840 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:09.538408 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:09.538503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:09.539416 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:09.539423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:09.539436 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:09.540111 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:09.540155 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:09.540176 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:09.540269 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:09.540461 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 06:47:09.540571 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260809 06:47:09.542194 10830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:09.621728 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 10639
I20260809 06:47:09.641419 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 10104
I20260809 06:47:09.642697 10204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:09.743981 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 10104
2026-08-09T06:47:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2453 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T06:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:47:09Z Disabled control of system clock
I20260809 06:47:09.827382 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37457
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:37457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 06:47:10.153622 10871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:10.153915 10871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:10.154043 10871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:10.161159 10871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:47:10.161259 10871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:10.161321 10871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:10.161398 10871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:47:10.161458 10871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:47:10.170285 10871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:37457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:10.171499 10871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:10.173326 10871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:10.185108 10880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:10.185164 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:10.185086 10882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:10.185843 10871 server_base.cc:1034] running on GCE node
I20260809 06:47:10.186767 10871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:10.188400 10871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:10.189626 10871 hybrid_clock.cc:648] HybridClock initialized: now 1786258030189568 us; error 39 us; skew 500 ppm
I20260809 06:47:10.190035 10871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:10.192309 10871 webserver.cc:460] Webserver started at http://127.27.209.254:39833/ using document root <none> and password file <none>
I20260809 06:47:10.193007 10871 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:10.193112 10871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:10.193431 10871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:10.196364 10871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "9fb60726322d4923832ee0223ff6f179"
format_stamp: "Formatted at 2026-08-09 06:47:10 on dist-test-slave-jpj1"
server_key: "41b53126d21c4b570d705bb6522ba61b"
server_key_iv: "5d72b1a653a4034f8fbdb86eb51fb2d9"
server_key_version: "encryptionkey@0"
I20260809 06:47:10.197166 10871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "9fb60726322d4923832ee0223ff6f179"
format_stamp: "Formatted at 2026-08-09 06:47:10 on dist-test-slave-jpj1"
server_key: "41b53126d21c4b570d705bb6522ba61b"
server_key_iv: "5d72b1a653a4034f8fbdb86eb51fb2d9"
server_key_version: "encryptionkey@0"
I20260809 06:47:10.202138 10871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:47:10.205478 10897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:10.206745 10871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:47:10.206903 10871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "9fb60726322d4923832ee0223ff6f179"
format_stamp: "Formatted at 2026-08-09 06:47:10 on dist-test-slave-jpj1"
server_key: "41b53126d21c4b570d705bb6522ba61b"
server_key_iv: "5d72b1a653a4034f8fbdb86eb51fb2d9"
server_key_version: "encryptionkey@0"
I20260809 06:47:10.207103 10871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:10.212610 10871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:10.213413 10871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:10.213692 10871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:10.230647 10871 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:37457
I20260809 06:47:10.230683 10997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:37457 every 8 connection(s)
I20260809 06:47:10.231930 10871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:47:10.235814 10998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:47:10.238415 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 10871
I20260809 06:47:10.238624 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:47:10.239274 28487 external_mini_cluster.cc:1428] Setting key 6b9f1b0cf836617d275a719c78018c31
I20260809 06:47:10.247440 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:10.241288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.27.209.254:37457)
0809 06:47:10.242217 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:10.242227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:10.242270 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:47:10.243972 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:10.243996 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:10.244445 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:10.244777 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:10.245876 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:10.245882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:10.245896 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:10.245910 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:10.245973 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:10.245998 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:10.246179 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:10.246296 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 06:47:10.246536 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":331,"threads_started":1}
I20260809 06:47:10.247953 10998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179: Bootstrap starting.
I20260809 06:47:10.251026 10998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179: Neither blocks nor log segments found. Creating new log.
I20260809 06:47:10.252049 10998 log.cc:824] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179: Log is configured to *not* fsync() on all Append() calls
I20260809 06:47:10.254699 10998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179: No bootstrap required, opened a new log
I20260809 06:47:10.261765 10998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb60726322d4923832ee0223ff6f179" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37457 } }
I20260809 06:47:10.262120 10998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:47:10.262224 10998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb60726322d4923832ee0223ff6f179, State: Initialized, Role: FOLLOWER
I20260809 06:47:10.262768 10998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb60726322d4923832ee0223ff6f179" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37457 } }
I20260809 06:47:10.262991 10998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:47:10.263118 10998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:47:10.263278 10998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:47:10.265426 10998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb60726322d4923832ee0223ff6f179" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37457 } }
I20260809 06:47:10.265980 10998 leader_election.cc:302] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fb60726322d4923832ee0223ff6f179; no voters:
I20260809 06:47:10.266670 10998 leader_election.cc:288] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:47:10.266777 11006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:47:10.267442 11006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [term 1 LEADER]: Becoming Leader. State: Replica: 9fb60726322d4923832ee0223ff6f179, State: Running, Role: LEADER
I20260809 06:47:10.268033 11006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb60726322d4923832ee0223ff6f179" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37457 } }
I20260809 06:47:10.269032 10998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:47:10.270830 11008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb60726322d4923832ee0223ff6f179. Latest consensus state: current_term: 1 leader_uuid: "9fb60726322d4923832ee0223ff6f179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb60726322d4923832ee0223ff6f179" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37457 } } }
I20260809 06:47:10.271248 11008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [sys.catalog]: This master's current role is: LEADER
I20260809 06:47:10.271093 11007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fb60726322d4923832ee0223ff6f179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb60726322d4923832ee0223ff6f179" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37457 } } }
I20260809 06:47:10.271447 11007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179 [sys.catalog]: This master's current role is: LEADER
I20260809 06:47:10.273479 11016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:47:10.278383 11016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:47:10.287319 11016 catalog_manager.cc:1347] Generated new cluster ID: 47efa9dd5fea406d80083d94e7f9270d
I20260809 06:47:10.287416 11016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:47:10.309770 11016 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:47:10.310220 11016 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:47:10.316715 11016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fb60726322d4923832ee0223ff6f179: Generated new TSK 0
I20260809 06:47:10.317502 11016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:47:10.327325 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37457
--builtin_ntp_servers=127.27.209.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 06:47:10.680984 11032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:10.681275 11032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:10.681389 11032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:10.688539 11032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:10.688637 11032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:10.688776 11032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:47:10.698068 11032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:10.699301 11032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:10.701067 11032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:10.713768 11043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:10.713809 11039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:10.713729 11040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:10.714718 11032 server_base.cc:1034] running on GCE node
I20260809 06:47:10.715308 11032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:10.716233 11032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:10.717427 11032 hybrid_clock.cc:648] HybridClock initialized: now 1786258030717373 us; error 50 us; skew 500 ppm
I20260809 06:47:10.717726 11032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:10.720139 11032 webserver.cc:460] Webserver started at http://127.27.209.193:43477/ using document root <none> and password file <none>
I20260809 06:47:10.720806 11032 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:10.720908 11032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:10.721222 11032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:10.724184 11032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "9ba265ec758c4314bbbab185ed2d0cc3"
format_stamp: "Formatted at 2026-08-09 06:47:10 on dist-test-slave-jpj1"
server_key: "a117c32f23f69fd6c181088122128019"
server_key_iv: "02846de5a9173c10a11fb9b5ba6fd98a"
server_key_version: "encryptionkey@0"
I20260809 06:47:10.724973 11032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "9ba265ec758c4314bbbab185ed2d0cc3"
format_stamp: "Formatted at 2026-08-09 06:47:10 on dist-test-slave-jpj1"
server_key: "a117c32f23f69fd6c181088122128019"
server_key_iv: "02846de5a9173c10a11fb9b5ba6fd98a"
server_key_version: "encryptionkey@0"
I20260809 06:47:10.730078 11032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:47:10.733469 11049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:10.734781 11032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:47:10.734934 11032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "9ba265ec758c4314bbbab185ed2d0cc3"
format_stamp: "Formatted at 2026-08-09 06:47:10 on dist-test-slave-jpj1"
server_key: "a117c32f23f69fd6c181088122128019"
server_key_iv: "02846de5a9173c10a11fb9b5ba6fd98a"
server_key_version: "encryptionkey@0"
I20260809 06:47:10.735129 11032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:10.740665 11032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:10.741417 11032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:10.741658 11032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:10.742545 11032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:10.743971 11032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:10.744069 11032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:10.744184 11032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:10.744256 11032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:10.774025 11032 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:37977
I20260809 06:47:10.774130 11224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:37977 every 8 connection(s)
I20260809 06:47:10.775327 11032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:47:10.776808 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 11032
I20260809 06:47:10.777000 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:47:10.777712 28487 external_mini_cluster.cc:1428] Setting key 8b3de90509dcb5fcebab22ab0838aa33
I20260809 06:47:10.780686 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37457
--builtin_ntp_servers=127.27.209.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 06:47:10.783993 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:10.778032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:38685 (local address 127.27.209.254:37457)
0809 06:47:10.778656 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:10.778663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:10.779982 (+ 1319us) server_negotiation.cc:407] Connection header received
0809 06:47:10.781809 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:10.781815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:10.781877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:10.781976 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:10.783248 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:10.783254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:10.783259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:10.783277 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:10.783321 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:10.783334 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:10.783439 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:10.783652 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 06:47:10.783717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":379,"threads_started":1}
I20260809 06:47:10.784929 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:10.778758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37457 (local address 127.27.209.193:38685)
0809 06:47:10.779821 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:10.779870 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:10.781546 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:10.782230 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:10.782239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:10.782329 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:10.782958 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:47:10.782960 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:10.782997 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:47:10.783082 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:10.783422 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:10.783432 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:10.783440 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:47:10.783773 (+ 333us) client_negotiation.cc:240] Negotiation successful
0809 06:47:10.784029 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260809 06:47:10.786383 11225 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37457
I20260809 06:47:10.786643 11225 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:10.787180 11225 heartbeater.cc:511] Master 127.27.209.254:37457 requested a full tablet report, sending...
I20260809 06:47:10.789321 10924 ts_manager.cc:194] Registered new tserver with Master: 9ba265ec758c4314bbbab185ed2d0cc3 (127.27.209.193:37977)
W20260809 06:47:11.105284 11230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:11.105552 11230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:11.105640 11230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:11.112703 11230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:11.112794 11230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:11.112908 11230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:47:11.121701 11230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:11.122900 11230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:11.124666 11230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:11.137161 11238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:11.137133 11236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:11.137765 11240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:11.137799 11230 server_base.cc:1034] running on GCE node
I20260809 06:47:11.138494 11230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:11.139412 11230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:11.140647 11230 hybrid_clock.cc:648] HybridClock initialized: now 1786258031140597 us; error 48 us; skew 500 ppm
I20260809 06:47:11.140933 11230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:11.142998 11230 webserver.cc:460] Webserver started at http://127.27.209.194:44753/ using document root <none> and password file <none>
I20260809 06:47:11.143689 11230 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:11.143784 11230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:11.144057 11230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:11.146883 11230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "27694593d3664c0e990e06e195314c5a"
format_stamp: "Formatted at 2026-08-09 06:47:11 on dist-test-slave-jpj1"
server_key: "801da6ff8b09d1e28ee395ca7329f30b"
server_key_iv: "7cbd8956d1d4f6ea0e848a1e2fc65ad8"
server_key_version: "encryptionkey@0"
I20260809 06:47:11.147600 11230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "27694593d3664c0e990e06e195314c5a"
format_stamp: "Formatted at 2026-08-09 06:47:11 on dist-test-slave-jpj1"
server_key: "801da6ff8b09d1e28ee395ca7329f30b"
server_key_iv: "7cbd8956d1d4f6ea0e848a1e2fc65ad8"
server_key_version: "encryptionkey@0"
I20260809 06:47:11.152505 11230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260809 06:47:11.155743 11247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:11.156996 11230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:47:11.157146 11230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "27694593d3664c0e990e06e195314c5a"
format_stamp: "Formatted at 2026-08-09 06:47:11 on dist-test-slave-jpj1"
server_key: "801da6ff8b09d1e28ee395ca7329f30b"
server_key_iv: "7cbd8956d1d4f6ea0e848a1e2fc65ad8"
server_key_version: "encryptionkey@0"
I20260809 06:47:11.157336 11230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:11.162688 11230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:11.163410 11230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:11.163686 11230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:11.164548 11230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:11.165906 11230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:11.166002 11230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:11.166108 11230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:11.166180 11230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:11.196353 11230 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:36929
I20260809 06:47:11.196385 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:36929 every 8 connection(s)
I20260809 06:47:11.197664 11230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:47:11.205354 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.200196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:46349 (local address 127.27.209.254:37457)
0809 06:47:11.200384 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:11.200389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:11.201749 (+ 1360us) server_negotiation.cc:407] Connection header received
0809 06:47:11.203367 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:11.203372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:11.203426 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:11.203512 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:11.204782 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:11.204787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:11.204790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:11.204807 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:11.204840 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:11.204852 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:11.204943 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:11.205090 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 06:47:11.205139 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:47:11.206254 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.200706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37457 (local address 127.27.209.194:46349)
0809 06:47:11.201586 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:11.201622 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:11.203132 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:11.203770 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:11.203779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:11.203869 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:11.204471 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:47:11.204473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:11.204511 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:47:11.204601 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:11.204931 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:11.204938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:11.204943 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:47:11.205150 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 06:47:11.205337 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":253,"threads_started":1}
I20260809 06:47:11.206712 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 11230
I20260809 06:47:11.206899 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:47:11.207399 28487 external_mini_cluster.cc:1428] Setting key aa378cd5a123fbc8a4c9bfe05903d921
I20260809 06:47:11.207520 11427 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37457
I20260809 06:47:11.207824 11427 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:11.208508 11427 heartbeater.cc:511] Master 127.27.209.254:37457 requested a full tablet report, sending...
I20260809 06:47:11.209975 10924 ts_manager.cc:194] Registered new tserver with Master: 27694593d3664c0e990e06e195314c5a (127.27.209.194:36929)
I20260809 06:47:11.210274 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37457
--builtin_ntp_servers=127.27.209.212:35661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 06:47:11.538177 11432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:11.538498 11432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:11.538637 11432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:11.545930 11432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:11.546030 11432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:11.546176 11432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:47:11.555109 11432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:11.556403 11432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:11.558230 11432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:11.571183 11444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:11.571201 11440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:11.571254 11447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:11.576931 11432 server_base.cc:1034] running on GCE node
I20260809 06:47:11.577565 11432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:11.578524 11432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:11.579720 11432 hybrid_clock.cc:648] HybridClock initialized: now 1786258031579673 us; error 42 us; skew 500 ppm
I20260809 06:47:11.580026 11432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:47:11.582191 11432 webserver.cc:460] Webserver started at http://127.27.209.195:34785/ using document root <none> and password file <none>
I20260809 06:47:11.582885 11432 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:11.582978 11432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:11.583285 11432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:11.586203 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "dae956644e8548c2b97b6f77e0f28696"
format_stamp: "Formatted at 2026-08-09 06:47:11 on dist-test-slave-jpj1"
server_key: "44467fb336ac3cb4b6a7763374d10b23"
server_key_iv: "46dc7b96025973cdb43a0924d57fc804"
server_key_version: "encryptionkey@0"
I20260809 06:47:11.586971 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "dae956644e8548c2b97b6f77e0f28696"
format_stamp: "Formatted at 2026-08-09 06:47:11 on dist-test-slave-jpj1"
server_key: "44467fb336ac3cb4b6a7763374d10b23"
server_key_iv: "46dc7b96025973cdb43a0924d57fc804"
server_key_version: "encryptionkey@0"
I20260809 06:47:11.592057 11432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 06:47:11.595345 11457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:11.596648 11432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:47:11.596808 11432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "dae956644e8548c2b97b6f77e0f28696"
format_stamp: "Formatted at 2026-08-09 06:47:11 on dist-test-slave-jpj1"
server_key: "44467fb336ac3cb4b6a7763374d10b23"
server_key_iv: "46dc7b96025973cdb43a0924d57fc804"
server_key_version: "encryptionkey@0"
I20260809 06:47:11.597025 11432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:11.602613 11432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:11.603374 11432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:11.603650 11432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:11.604537 11432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:11.606006 11432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:11.606101 11432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:11.606212 11432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:11.606292 11432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:11.636632 11432 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:37049
I20260809 06:47:11.636703 11623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:37049 every 8 connection(s)
I20260809 06:47:11.637898 11432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:47:11.645704 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.640464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:42247 (local address 127.27.209.254:37457)
0809 06:47:11.640690 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:11.640697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:11.642066 (+ 1369us) server_negotiation.cc:407] Connection header received
0809 06:47:11.643755 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:11.643759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:11.643813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:11.643902 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:11.645149 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:11.645153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:11.645156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:11.645171 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:11.645203 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:11.645215 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:11.645310 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:11.645443 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 06:47:11.645491 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:47:11.646274 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 11432
I20260809 06:47:11.646435 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:47:11.646644 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.641004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37457 (local address 127.27.209.195:42247)
0809 06:47:11.641910 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:11.641955 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:11.643456 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:11.644149 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:11.644166 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:11.644257 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:11.644872 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:47:11.644875 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:11.644911 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:47:11.644995 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:11.645293 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:11.645299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:11.645304 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:47:11.645504 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 06:47:11.645690 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":270,"threads_started":1}
I20260809 06:47:11.646957 28487 external_mini_cluster.cc:1428] Setting key 6e6c55991c86169e9c8d5c195efb2109
I20260809 06:47:11.648926 11624 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37457
I20260809 06:47:11.649241 11624 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:11.649926 11624 heartbeater.cc:511] Master 127.27.209.254:37457 requested a full tablet report, sending...
I20260809 06:47:11.651213 10924 ts_manager.cc:194] Registered new tserver with Master: dae956644e8548c2b97b6f77e0f28696 (127.27.209.195:37049)
I20260809 06:47:11.662266 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 06:47:11.668062 11228 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:47:11.666619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.27.209.254:37457)
0809 06:47:11.666805 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:11.666811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:11.667277 (+ 466us) server_negotiation.cc:407] Connection header received
0809 06:47:11.667401 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:11.667405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:11.667455 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:11.667562 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:11.667875 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54752: BlockingRecv error: recv got EOF from 127.0.0.1:54752 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:47:11.668519 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 11032
I20260809 06:47:11.672036 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.669575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.27.209.193:37977)
0809 06:47:11.670109 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:11.670129 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:11.670160 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:47:11.670221 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:11.670226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:11.670425 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:11.670539 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:11.671371 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:11.671377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:11.671389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:11.671407 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:11.671445 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:11.671459 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:11.671585 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:11.671657 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 06:47:11.671749 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":233,"threads_started":1}
I20260809 06:47:11.673352 11214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:11.752744 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 11032
I20260809 06:47:11.775494 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 11230
I20260809 06:47:11.778930 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.776470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.27.209.194:36929)
0809 06:47:11.776983 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:11.776990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:11.777017 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:47:11.777087 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:11.777094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:11.777328 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:11.777456 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:11.778358 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:11.778364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:11.778375 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:11.778394 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:11.778435 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:11.778449 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:11.778555 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:11.778607 (+ 52us) server_negotiation.cc:299] Negotiation successful
0809 06:47:11.778691 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":230,"threads_started":1}
I20260809 06:47:11.780336 11417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:11.859498 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 11230
I20260809 06:47:11.881982 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 11432
I20260809 06:47:11.885998 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:11.883073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.27.209.195:37049)
0809 06:47:11.883802 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:11.883808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:11.883841 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:47:11.883907 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:11.883914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:11.884141 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:11.884268 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:11.885291 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:11.885298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:11.885310 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:47:11.885327 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:11.885367 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:47:11.885380 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:11.885514 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:11.885567 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 06:47:11.885661 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":374,"threads_started":1}
I20260809 06:47:11.887223 11617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:11.967036 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 11432
I20260809 06:47:11.987190 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 10871
I20260809 06:47:11.988350 10985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:12.090144 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 10871
2026-08-09T06:47:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2344 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:47:12 dist-test-slave-jpj1 krb5kdc[11653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:47:12 dist-test-slave-jpj1 krb5kdc[11653](info): set up 2 sockets
Aug 09 06:47:12 dist-test-slave-jpj1 krb5kdc[11653](info): commencing operation
krb5kdc: starting...
W20260809 06:47:13.297363 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:47:13 dist-test-slave-jpj1 krb5kdc[11653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:47:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:47:13.551607 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38711
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:38711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:47:13.923277 11672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:13.924213 11672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:13.924484 11672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:13.943468 11672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:47:13.943794 11672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:13.944118 11672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:13.944235 11672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:47:13.944375 11672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:47:13.972568 11672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:38711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:13.976554 11672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:13.980525 11672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:14.012060 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:14.012143 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:14.012384 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:14.015303 11672 server_base.cc:1034] running on GCE node
I20260809 06:47:14.017690 11672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:14.021385 11672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:14.024943 11672 hybrid_clock.cc:648] HybridClock initialized: now 1786258034024742 us; error 105 us; skew 500 ppm
I20260809 06:47:14.025980 11672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:47:14 dist-test-slave-jpj1 krb5kdc[11653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:47:14.038190 11672 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:47:14.041822 11672 webserver.cc:460] Webserver started at http://127.27.209.254:45821/ using document root <none> and password file <none>
I20260809 06:47:14.043437 11672 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:14.043874 11672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:14.044571 11672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:14.053699 11672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "001b124d91f3453fa5b1cc805c823782"
format_stamp: "Formatted at 2026-08-09 06:47:14 on dist-test-slave-jpj1"
server_key: "14121570e82ceb7c305b096e235f820b"
server_key_iv: "48518a34f17b1e73bf474751a6affe2b"
server_key_version: "encryptionkey@0"
I20260809 06:47:14.056296 11672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "001b124d91f3453fa5b1cc805c823782"
format_stamp: "Formatted at 2026-08-09 06:47:14 on dist-test-slave-jpj1"
server_key: "14121570e82ceb7c305b096e235f820b"
server_key_iv: "48518a34f17b1e73bf474751a6affe2b"
server_key_version: "encryptionkey@0"
I20260809 06:47:14.070231 11672 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260809 06:47:14.081776 11693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:14.085670 11672 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260809 06:47:14.086164 11672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "001b124d91f3453fa5b1cc805c823782"
format_stamp: "Formatted at 2026-08-09 06:47:14 on dist-test-slave-jpj1"
server_key: "14121570e82ceb7c305b096e235f820b"
server_key_iv: "48518a34f17b1e73bf474751a6affe2b"
server_key_version: "encryptionkey@0"
I20260809 06:47:14.086817 11672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:14.132840 11672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:14.139503 11672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:14.139987 11672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:14.166172 11672 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:38711
I20260809 06:47:14.166205 11766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:38711 every 8 connection(s)
I20260809 06:47:14.167550 11672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:47:14.171326 11767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:47:14.171330 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 11672
I20260809 06:47:14.171517 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:47:14.172190 28487 external_mini_cluster.cc:1428] Setting key 3e383f5ac206c1561a7123440975a821
I20260809 06:47:14.184762 11767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782: Bootstrap starting.
Aug 09 06:47:14 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:47:14.188339 11767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782: Neither blocks nor log segments found. Creating new log.
I20260809 06:47:14.189529 11767 log.cc:824] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782: Log is configured to *not* fsync() on all Append() calls
I20260809 06:47:14.192658 11767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782: No bootstrap required, opened a new log
I20260809 06:47:14.195776 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:14.174112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.27.209.254:38711)
0809 06:47:14.174890 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:14.174905 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:14.174949 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:47:14.177107 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:14.177132 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:14.177529 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:14.177862 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:14.179695 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:14.180937 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:14.182771 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:14.183418 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:14.183570 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:14.188250 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:14.188258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:14.188273 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:14.188309 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:14.191759 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:14.192421 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:14.192428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:14.192434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:14.192604 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:14.193010 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:14.193014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:14.193016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:14.193540 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:14.193865 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:14.194236 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 06:47:14.194679 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":345,"threads_started":1}
I20260809 06:47:14.201395 11767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001b124d91f3453fa5b1cc805c823782" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38711 } }
I20260809 06:47:14.201767 11767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:47:14.201859 11767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b124d91f3453fa5b1cc805c823782, State: Initialized, Role: FOLLOWER
I20260809 06:47:14.202383 11767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001b124d91f3453fa5b1cc805c823782" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38711 } }
I20260809 06:47:14.202600 11767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:47:14.202732 11767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:47:14.202880 11767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:47:14.205178 11767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001b124d91f3453fa5b1cc805c823782" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38711 } }
I20260809 06:47:14.205652 11767 leader_election.cc:302] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 001b124d91f3453fa5b1cc805c823782; no voters:
I20260809 06:47:14.206274 11767 leader_election.cc:288] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:47:14.206420 11776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:47:14.207438 11776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [term 1 LEADER]: Becoming Leader. State: Replica: 001b124d91f3453fa5b1cc805c823782, State: Running, Role: LEADER
I20260809 06:47:14.208033 11776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001b124d91f3453fa5b1cc805c823782" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38711 } }
I20260809 06:47:14.208644 11767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:47:14.210898 11777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "001b124d91f3453fa5b1cc805c823782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001b124d91f3453fa5b1cc805c823782" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38711 } } }
I20260809 06:47:14.210999 11781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001b124d91f3453fa5b1cc805c823782. Latest consensus state: current_term: 1 leader_uuid: "001b124d91f3453fa5b1cc805c823782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001b124d91f3453fa5b1cc805c823782" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38711 } } }
I20260809 06:47:14.211288 11777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [sys.catalog]: This master's current role is: LEADER
I20260809 06:47:14.211437 11781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782 [sys.catalog]: This master's current role is: LEADER
I20260809 06:47:14.212986 11787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:47:14.218230 11787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:47:14.226974 11787 catalog_manager.cc:1347] Generated new cluster ID: 94b63c38f0204f0f85032dc9f70f2f29
I20260809 06:47:14.227087 11787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:47:14.247929 11787 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:47:14.249508 11787 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:47:14.258026 11787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782: Generated new TSK 0
I20260809 06:47:14.258812 11787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:47:14.323815 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38711
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:47:14.658119 11810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:14.658437 11810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:14.658574 11810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:14.665750 11810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:14.665870 11810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:14.666029 11810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:47:14.675611 11810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:14.676896 11810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:14.678190 11810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:14.691131 11815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:14.691085 11818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:14.691339 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:14.691761 11810 server_base.cc:1034] running on GCE node
I20260809 06:47:14.692306 11810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:14.693148 11810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:14.694337 11810 hybrid_clock.cc:648] HybridClock initialized: now 1786258034694296 us; error 34 us; skew 500 ppm
I20260809 06:47:14.694654 11810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:47:14 dist-test-slave-jpj1 krb5kdc[11653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:47:14.697793 11810 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:47:14.699221 11810 webserver.cc:460] Webserver started at http://127.27.209.193:33827/ using document root <none> and password file <none>
I20260809 06:47:14.699928 11810 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:14.700048 11810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:14.700340 11810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:14.703166 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "fdddf394ed094c77a25f40fb584a913d"
format_stamp: "Formatted at 2026-08-09 06:47:14 on dist-test-slave-jpj1"
server_key: "b3d29c20aa58ab4703f2c1b5e88fb50c"
server_key_iv: "f4d0d7e0e6a0f9162177aed8c4b76b11"
server_key_version: "encryptionkey@0"
I20260809 06:47:14.704064 11810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "fdddf394ed094c77a25f40fb584a913d"
format_stamp: "Formatted at 2026-08-09 06:47:14 on dist-test-slave-jpj1"
server_key: "b3d29c20aa58ab4703f2c1b5e88fb50c"
server_key_iv: "f4d0d7e0e6a0f9162177aed8c4b76b11"
server_key_version: "encryptionkey@0"
I20260809 06:47:14.709403 11810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260809 06:47:14.712896 11831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:14.714196 11810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:47:14.714358 11810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "fdddf394ed094c77a25f40fb584a913d"
format_stamp: "Formatted at 2026-08-09 06:47:14 on dist-test-slave-jpj1"
server_key: "b3d29c20aa58ab4703f2c1b5e88fb50c"
server_key_iv: "f4d0d7e0e6a0f9162177aed8c4b76b11"
server_key_version: "encryptionkey@0"
I20260809 06:47:14.714537 11810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:14.742018 11810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:14.745985 11810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:14.746281 11810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:14.747282 11810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:14.748669 11810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:14.748750 11810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:14.748875 11810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:14.748930 11810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:14.779174 11810 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:42687
I20260809 06:47:14.779227 11998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:42687 every 8 connection(s)
I20260809 06:47:14.780455 11810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:47:14.781531 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 11810
I20260809 06:47:14.781773 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:47:14.782636 28487 external_mini_cluster.cc:1428] Setting key 99f8b60a8072816d29d8eb9fc2a59f26
Aug 09 06:47:14 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:47:14.801934 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:14.783264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:33387 (local address 127.27.209.254:38711)
0809 06:47:14.783755 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:14.783760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:14.784830 (+ 1070us) server_negotiation.cc:407] Connection header received
0809 06:47:14.787263 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:14.787267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:14.787330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:14.787415 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:14.789309 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:14.790053 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:14.792093 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:14.792503 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:14.792602 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:14.796479 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:14.796484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:14.796487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:14.796520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:14.798921 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:14.799637 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:14.799641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:14.799643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:14.799783 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:14.800221 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:14.800225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:14.800227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:14.800498 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:14.800632 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:14.801329 (+ 697us) server_negotiation.cc:299] Negotiation successful
0809 06:47:14.801560 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260809 06:47:14.802899 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:14.783859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38711 (local address 127.27.209.193:33387)
0809 06:47:14.784681 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:14.784724 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:14.786958 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:14.787691 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:14.787703 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:14.788396 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:14.789122 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:14.789147 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:14.790222 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:14.790227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:14.791950 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:14.791971 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:14.792726 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:14.792732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:14.793012 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:14.793750 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:14.793779 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:14.796236 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:14.799052 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:47:14.799062 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:47:14.799078 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:47:14.799504 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:47:14.799888 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:47:14.799893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:47:14.799897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:47:14.800096 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:47:14.800621 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:14.800628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:14.800965 (+ 337us) client_negotiation.cc:769] Sending connection context
0809 06:47:14.801200 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 06:47:14.801739 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":322,"threads_started":1}
I20260809 06:47:14.804483 11999 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38711
I20260809 06:47:14.804848 11999 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:14.805727 11999 heartbeater.cc:511] Master 127.27.209.254:38711 requested a full tablet report, sending...
I20260809 06:47:14.808246 11716 ts_manager.cc:194] Registered new tserver with Master: fdddf394ed094c77a25f40fb584a913d (127.27.209.193:42687)
I20260809 06:47:14.810427 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:33387
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:47:14.844770 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38711
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:47:15.186251 12008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:15.186533 12008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:15.186677 12008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:15.194332 12008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:15.194448 12008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:15.194568 12008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:47:15.204440 12008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:15.205726 12008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:15.206980 12008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:15.219853 12015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:15.219974 12018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:15.220000 12016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:15.221343 12008 server_base.cc:1034] running on GCE node
I20260809 06:47:15.221961 12008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:15.223006 12008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:15.224206 12008 hybrid_clock.cc:648] HybridClock initialized: now 1786258035224155 us; error 50 us; skew 500 ppm
I20260809 06:47:15.224531 12008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:47:15 dist-test-slave-jpj1 krb5kdc[11653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:47:15.227677 12008 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:47:15.229044 12008 webserver.cc:460] Webserver started at http://127.27.209.194:43115/ using document root <none> and password file <none>
I20260809 06:47:15.229730 12008 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:15.229851 12008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:15.230139 12008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:15.233047 12008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "f4c29759cc124693b9dd46151ada086e"
format_stamp: "Formatted at 2026-08-09 06:47:15 on dist-test-slave-jpj1"
server_key: "2654fee09d1e4e3a728ab71c134c9635"
server_key_iv: "15b7eff9c41456e0ac34aaf504e66232"
server_key_version: "encryptionkey@0"
I20260809 06:47:15.233798 12008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "f4c29759cc124693b9dd46151ada086e"
format_stamp: "Formatted at 2026-08-09 06:47:15 on dist-test-slave-jpj1"
server_key: "2654fee09d1e4e3a728ab71c134c9635"
server_key_iv: "15b7eff9c41456e0ac34aaf504e66232"
server_key_version: "encryptionkey@0"
I20260809 06:47:15.238875 12008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:47:15.242182 12026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:15.243484 12008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:47:15.243656 12008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "f4c29759cc124693b9dd46151ada086e"
format_stamp: "Formatted at 2026-08-09 06:47:15 on dist-test-slave-jpj1"
server_key: "2654fee09d1e4e3a728ab71c134c9635"
server_key_iv: "15b7eff9c41456e0ac34aaf504e66232"
server_key_version: "encryptionkey@0"
I20260809 06:47:15.243839 12008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:15.261718 12008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:15.265580 12008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:15.265849 12008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:15.266769 12008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:15.268090 12008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:15.268172 12008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:15.268301 12008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:15.268353 12008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:15.298398 12008 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:45757
I20260809 06:47:15.298440 12197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:45757 every 8 connection(s)
I20260809 06:47:15.299646 12008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:47:15.300143 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 12008
I20260809 06:47:15.300323 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:47:15.300930 28487 external_mini_cluster.cc:1428] Setting key 0c7ed4cab734641058a09d363966bc1f
Aug 09 06:47:15 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:47:15.319703 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.302201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:57847 (local address 127.27.209.254:38711)
0809 06:47:15.302412 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:15.302418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:15.303737 (+ 1319us) server_negotiation.cc:407] Connection header received
0809 06:47:15.305608 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:15.305613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:15.305696 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:15.305813 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:15.307252 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.308176 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.309783 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.310350 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.310473 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.313827 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:15.313834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:15.313839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:15.313880 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:15.316569 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:15.317390 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:15.317395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:15.317398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:15.317526 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:15.318021 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:15.318025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:15.318027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:15.318288 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:15.318400 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:15.319200 (+ 800us) server_negotiation.cc:299] Negotiation successful
0809 06:47:15.319410 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:47:15.321058 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.302756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38711 (local address 127.27.209.194:57847)
0809 06:47:15.303592 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:15.303639 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:15.305347 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:15.306019 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:15.306027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:15.306545 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:15.307081 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.307098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.308324 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.308327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.309626 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.309639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.310541 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.310546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.310755 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.311382 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:15.311408 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:15.313563 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:15.316714 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:47:15.316723 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:47:15.316738 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:47:15.317275 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:47:15.317634 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:47:15.317638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:47:15.317641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:47:15.317906 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:47:15.318408 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:15.318416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:15.318816 (+ 400us) client_negotiation.cc:769] Sending connection context
0809 06:47:15.319109 (+ 293us) client_negotiation.cc:240] Negotiation successful
0809 06:47:15.319727 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":316,"threads_started":1}
I20260809 06:47:15.322625 12200 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38711
I20260809 06:47:15.322943 12200 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:15.323791 12200 heartbeater.cc:511] Master 127.27.209.254:38711 requested a full tablet report, sending...
I20260809 06:47:15.325292 11716 ts_manager.cc:194] Registered new tserver with Master: f4c29759cc124693b9dd46151ada086e (127.27.209.194:45757)
I20260809 06:47:15.326463 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:57847
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:47:15.361991 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38711
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:47:15.688467 12207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:15.688747 12207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:15.688884 12207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:15.696004 12207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:15.696111 12207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:15.696271 12207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:47:15.705773 12207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:15.707036 12207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:15.708415 12207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:15.721268 12213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:15.721316 12214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:15.721211 12219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:15.722216 12207 server_base.cc:1034] running on GCE node
I20260809 06:47:15.722836 12207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:15.723810 12207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:15.725021 12207 hybrid_clock.cc:648] HybridClock initialized: now 1786258035724973 us; error 43 us; skew 500 ppm
I20260809 06:47:15.725301 12207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:47:15 dist-test-slave-jpj1 krb5kdc[11653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:47:15.728433 12207 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:47:15.729861 12207 webserver.cc:460] Webserver started at http://127.27.209.195:36747/ using document root <none> and password file <none>
I20260809 06:47:15.730506 12207 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:15.730612 12207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:15.730875 12207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:47:15.733712 12207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "98082a206958458c8cdec165cdc3893c"
format_stamp: "Formatted at 2026-08-09 06:47:15 on dist-test-slave-jpj1"
server_key: "72afb89c56ed110d9e0499845b18e867"
server_key_iv: "a0bd38200589b19ad423e53bdeef167f"
server_key_version: "encryptionkey@0"
I20260809 06:47:15.734499 12207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "98082a206958458c8cdec165cdc3893c"
format_stamp: "Formatted at 2026-08-09 06:47:15 on dist-test-slave-jpj1"
server_key: "72afb89c56ed110d9e0499845b18e867"
server_key_iv: "a0bd38200589b19ad423e53bdeef167f"
server_key_version: "encryptionkey@0"
I20260809 06:47:15.739475 12207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:47:15.742933 12230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:15.744370 12207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:47:15.744506 12207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "98082a206958458c8cdec165cdc3893c"
format_stamp: "Formatted at 2026-08-09 06:47:15 on dist-test-slave-jpj1"
server_key: "72afb89c56ed110d9e0499845b18e867"
server_key_iv: "a0bd38200589b19ad423e53bdeef167f"
server_key_version: "encryptionkey@0"
I20260809 06:47:15.744669 12207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:47:15.755841 12207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:47:15.759791 12207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:47:15.760031 12207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:47:15.760875 12207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:47:15.762125 12207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:47:15.762197 12207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:15.762286 12207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:47:15.762329 12207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:47:15.792311 12207 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:33671
I20260809 06:47:15.792366 12404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:33671 every 8 connection(s)
I20260809 06:47:15.793491 12207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:47:15.795218 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 12207
I20260809 06:47:15.795428 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:47:15.795998 28487 external_mini_cluster.cc:1428] Setting key 588592b67cc73b27b42eb3ae7132c24d
Aug 09 06:47:15 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:47:15.813547 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.795962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:51003 (local address 127.27.209.254:38711)
0809 06:47:15.796135 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:15.796141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:15.797577 (+ 1436us) server_negotiation.cc:407] Connection header received
0809 06:47:15.799314 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:15.799318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:15.799380 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:15.799459 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:15.800924 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.801684 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.803689 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.804118 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.804217 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.807434 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:15.807439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:15.807442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:15.807474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:15.810305 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:15.811270 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:15.811276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:15.811279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:15.811437 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:15.811893 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:15.811897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:15.811899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:15.812186 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:15.812305 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:15.813083 (+ 778us) server_negotiation.cc:299] Negotiation successful
0809 06:47:15.813265 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:47:15.813673 11999 heartbeater.cc:503] Master 127.27.209.254:38711 was elected leader, sending a full tablet report...
I20260809 06:47:15.814713 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.796593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38711 (local address 127.27.209.195:51003)
0809 06:47:15.797409 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:15.797461 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:15.799067 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:15.799693 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:15.799704 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:15.800224 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:47:15.800775 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.800795 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.801827 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.801833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.803493 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.803510 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.804316 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.804322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.804525 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.805107 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:47:15.805128 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:47:15.807248 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:47:15.810451 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:47:15.810461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:47:15.810475 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:47:15.810986 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:47:15.811573 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:47:15.811578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:47:15.811583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:47:15.811784 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:47:15.812294 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:47:15.812312 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:47:15.812679 (+ 367us) client_negotiation.cc:769] Sending connection context
0809 06:47:15.812911 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 06:47:15.813310 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":316,"threads_started":1}
I20260809 06:47:15.816054 12405 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38711
I20260809 06:47:15.816367 12405 heartbeater.cc:464] Registering TS with master...
I20260809 06:47:15.817061 12405 heartbeater.cc:511] Master 127.27.209.254:38711 requested a full tablet report, sending...
I20260809 06:47:15.818464 11716 ts_manager.cc:194] Registered new tserver with Master: 98082a206958458c8cdec165cdc3893c (127.27.209.195:33671)
I20260809 06:47:15.819263 11716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:51003
I20260809 06:47:15.823310 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:47:15.839136 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.827684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.27.209.254:38711)
0809 06:47:15.827836 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:15.827841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:15.828291 (+ 450us) server_negotiation.cc:407] Connection header received
0809 06:47:15.828421 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:15.828425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:15.828481 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:15.828560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:15.830033 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.830798 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.832020 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.832424 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.832519 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.834731 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:15.834736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:15.834739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:15.834767 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:15.837091 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:15.837630 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:15.837634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:15.837636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:15.837770 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:15.838028 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:15.838031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:15.838033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:15.838333 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:15.838513 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:15.838768 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 06:47:15.838922 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 06:47:15.842784 11716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:47:15.849674 11716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:47:15.872960 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.862183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.27.209.194:45757)
0809 06:47:15.866012 (+ 3829us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:15.866019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:15.866043 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:47:15.866123 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:15.866127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:15.866327 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:15.867400 (+ 1073us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:47:15.867638 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.868618 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.870867 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.871837 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.871986 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.872164 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:15.872482 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 06:47:15.872639 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3692,"thread_start_us":3557,"threads_started":1}
I20260809 06:47:15.873003 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.861573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:45757 (local address 127.0.0.1:60508)
0809 06:47:15.863963 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:15.864006 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:15.864186 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:15.866572 (+ 2386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:15.866581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:15.866607 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:47:15.867150 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.867167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.868746 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.868752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.870716 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.870729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.872057 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.872064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.872262 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.872300 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 06:47:15.872408 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 06:47:15.872582 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"mutex_wait_us":27,"thread_start_us":294,"threads_started":1}
I20260809 06:47:15.874646 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.861757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33671 (local address 127.0.0.1:51782)
0809 06:47:15.864730 (+ 2973us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:15.864757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:15.864858 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:15.866895 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:15.866900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:15.866923 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:47:15.867455 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.867468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.869368 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.869373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.872010 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.872021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.873849 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.873856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.874011 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.874044 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:47:15.874125 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:47:15.874252 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2848,"mutex_wait_us":85,"thread_start_us":2707,"threads_started":1}
I20260809 06:47:15.875312 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.862677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.27.209.195:33671)
0809 06:47:15.866343 (+ 3666us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:15.866350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:15.866381 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:47:15.866471 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:15.866477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:15.866675 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:15.866881 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:47:15.867657 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.869211 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.872182 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.873657 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.873839 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.873968 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:15.874808 (+ 840us) server_negotiation.cc:299] Negotiation successful
0809 06:47:15.874976 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2873,"thread_start_us":2701,"threads_started":1}
I20260809 06:47:15.875313 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.862667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.27.209.193:42687)
0809 06:47:15.863277 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:15.863285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:15.865409 (+ 2124us) server_negotiation.cc:407] Connection header received
0809 06:47:15.865906 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:15.865914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:15.866189 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:15.866354 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:47:15.867450 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.869105 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.872182 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.873598 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.873770 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.873938 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:15.874516 (+ 578us) server_negotiation.cc:299] Negotiation successful
0809 06:47:15.874968 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":275,"threads_started":1}
I20260809 06:47:15.876003 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:15.862152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:42687 (local address 127.0.0.1:52486)
0809 06:47:15.865296 (+ 3144us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:15.865314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:15.865692 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:15.866624 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:15.866628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:15.866654 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:47:15.867257 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.867269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.869283 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.869288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.871999 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:15.872015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:15.874137 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:15.874144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:15.874312 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:15.874342 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:47:15.875496 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0809 06:47:15.875649 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3026,"mutex_wait_us":94,"spinlock_wait_cycles":4480,"thread_start_us":2909,"threads_started":1}
I20260809 06:47:15.877053 12103 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9816eee2fc4dcfa5dade6edc56271d (DEFAULT_TABLE table=test-table [id=339b86d1f7ff49e989d6834f8fc6a9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:47:15.878242 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9816eee2fc4dcfa5dade6edc56271d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:47:15.879257 12306 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9816eee2fc4dcfa5dade6edc56271d (DEFAULT_TABLE table=test-table [id=339b86d1f7ff49e989d6834f8fc6a9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:47:15.879617 11903 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9816eee2fc4dcfa5dade6edc56271d (DEFAULT_TABLE table=test-table [id=339b86d1f7ff49e989d6834f8fc6a9e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:47:15.880795 12306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9816eee2fc4dcfa5dade6edc56271d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:47:15.880903 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9816eee2fc4dcfa5dade6edc56271d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:47:15.889703 12437 tablet_bootstrap.cc:492] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Bootstrap starting.
I20260809 06:47:15.893177 12437 tablet_bootstrap.cc:654] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Neither blocks nor log segments found. Creating new log.
I20260809 06:47:15.894281 12437 log.cc:824] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:47:15.895078 12439 tablet_bootstrap.cc:492] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Bootstrap starting.
I20260809 06:47:15.895881 12438 tablet_bootstrap.cc:492] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Bootstrap starting.
I20260809 06:47:15.898610 12439 tablet_bootstrap.cc:654] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Neither blocks nor log segments found. Creating new log.
I20260809 06:47:15.899230 12438 tablet_bootstrap.cc:654] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Neither blocks nor log segments found. Creating new log.
I20260809 06:47:15.899881 12439 log.cc:824] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:47:15.900398 12438 log.cc:824] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Log is configured to *not* fsync() on all Append() calls
I20260809 06:47:15.901546 12437 tablet_bootstrap.cc:492] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: No bootstrap required, opened a new log
I20260809 06:47:15.901830 12437 ts_tablet_manager.cc:1397] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260809 06:47:15.902894 12439 tablet_bootstrap.cc:492] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: No bootstrap required, opened a new log
I20260809 06:47:15.903164 12439 ts_tablet_manager.cc:1397] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:47:15.904161 12438 tablet_bootstrap.cc:492] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: No bootstrap required, opened a new log
I20260809 06:47:15.904402 12438 ts_tablet_manager.cc:1397] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 06:47:15.910291 12439 raft_consensus.cc:348] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:15.910638 12439 raft_consensus.cc:374] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:47:15.910749 12439 raft_consensus.cc:729] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98082a206958458c8cdec165cdc3893c, State: Initialized, Role: FOLLOWER
I20260809 06:47:15.911321 12439 consensus_queue.cc:260] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:15.911723 12437 raft_consensus.cc:348] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:15.912128 12437 raft_consensus.cc:374] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:47:15.912252 12437 raft_consensus.cc:729] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c29759cc124693b9dd46151ada086e, State: Initialized, Role: FOLLOWER
I20260809 06:47:15.912897 12437 consensus_queue.cc:260] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:15.914083 12200 heartbeater.cc:503] Master 127.27.209.254:38711 was elected leader, sending a full tablet report...
I20260809 06:47:15.914651 12437 ts_tablet_manager.cc:1428] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260809 06:47:15.915625 12438 raft_consensus.cc:348] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:15.916020 12438 raft_consensus.cc:374] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:47:15.916119 12438 raft_consensus.cc:729] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdddf394ed094c77a25f40fb584a913d, State: Initialized, Role: FOLLOWER
I20260809 06:47:15.916697 12438 consensus_queue.cc:260] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:15.919341 12438 ts_tablet_manager.cc:1428] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260809 06:47:15.920270 12405 heartbeater.cc:503] Master 127.27.209.254:38711 was elected leader, sending a full tablet report...
I20260809 06:47:15.920778 12439 ts_tablet_manager.cc:1428] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260809 06:47:16.032970 12002 tablet.cc:2369] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:47:16.045025 12406 tablet.cc:2369] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:47:16.051776 12201 tablet.cc:2369] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:47:16.057411 12444 raft_consensus.cc:482] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:47:16.057682 12444 raft_consensus.cc:504] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:16.059360 12444 leader_election.cc:288] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98082a206958458c8cdec165cdc3893c (127.27.209.195:33671), fdddf394ed094c77a25f40fb584a913d (127.27.209.193:42687)
I20260809 06:47:16.068408 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.059799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:57885 (local address 127.27.209.193:42687)
0809 06:47:16.060029 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:16.060035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:16.060736 (+ 701us) server_negotiation.cc:407] Connection header received
0809 06:47:16.060942 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:16.060948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:16.061027 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:16.061119 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:47:16.061967 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.063012 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.065992 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.067194 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.067370 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.067434 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:16.068015 (+ 581us) server_negotiation.cc:299] Negotiation successful
0809 06:47:16.068144 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 06:47:16.068403 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.059825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:47585 (local address 127.27.209.195:33671)
0809 06:47:16.060019 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:16.060026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:16.060734 (+ 708us) server_negotiation.cc:407] Connection header received
0809 06:47:16.060942 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:16.060948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:16.061027 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:16.061118 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:47:16.061852 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.062987 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.065988 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.067183 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.067355 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.067431 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:16.068007 (+ 576us) server_negotiation.cc:299] Negotiation successful
0809 06:47:16.068144 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:47:16.068634 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.059694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:42687 (local address 127.27.209.194:57885)
0809 06:47:16.060600 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:16.060623 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:16.060805 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:16.061154 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:16.061159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:16.061193 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:47:16.061823 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:16.061835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.063114 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.063119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:16.065822 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:16.065832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.067469 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.067474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:16.067770 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.067874 (+ 104us) client_negotiation.cc:769] Sending connection context
0809 06:47:16.067966 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 06:47:16.068134 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":221,"spinlock_wait_cycles":13696,"thread_start_us":566,"threads_started":1}
I20260809 06:47:16.068657 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.059763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33671 (local address 127.27.209.194:47585)
0809 06:47:16.060632 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0809 06:47:16.060646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:47:16.060805 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:47:16.061154 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:47:16.061159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:47:16.061209 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:47:16.061686 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:16.061702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.063118 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.063122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:16.065800 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:47:16.065820 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.067463 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.067469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:47:16.067772 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.067874 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 06:47:16.067969 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 06:47:16.068134 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":300,"spinlock_wait_cycles":41216,"thread_start_us":482,"threads_started":1}
I20260809 06:47:16.069783 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9816eee2fc4dcfa5dade6edc56271d" candidate_uuid: "f4c29759cc124693b9dd46151ada086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdddf394ed094c77a25f40fb584a913d" is_pre_election: true
I20260809 06:47:16.069888 12335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9816eee2fc4dcfa5dade6edc56271d" candidate_uuid: "f4c29759cc124693b9dd46151ada086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98082a206958458c8cdec165cdc3893c" is_pre_election: true
I20260809 06:47:16.070305 11930 raft_consensus.cc:2454] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c29759cc124693b9dd46151ada086e in term 0.
I20260809 06:47:16.070379 12335 raft_consensus.cc:2454] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c29759cc124693b9dd46151ada086e in term 0.
I20260809 06:47:16.071061 12029 leader_election.cc:302] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 98082a206958458c8cdec165cdc3893c, f4c29759cc124693b9dd46151ada086e, fdddf394ed094c77a25f40fb584a913d; no voters:
I20260809 06:47:16.071513 12444 raft_consensus.cc:2781] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:47:16.071689 12444 raft_consensus.cc:482] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:47:16.071810 12444 raft_consensus.cc:3037] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:47:16.073951 12444 raft_consensus.cc:504] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:16.074554 12444 leader_election.cc:288] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [CANDIDATE]: Term 1 election: Requested vote from peers 98082a206958458c8cdec165cdc3893c (127.27.209.195:33671), fdddf394ed094c77a25f40fb584a913d (127.27.209.193:42687)
I20260809 06:47:16.074927 12335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9816eee2fc4dcfa5dade6edc56271d" candidate_uuid: "f4c29759cc124693b9dd46151ada086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98082a206958458c8cdec165cdc3893c"
I20260809 06:47:16.074985 11930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9816eee2fc4dcfa5dade6edc56271d" candidate_uuid: "f4c29759cc124693b9dd46151ada086e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdddf394ed094c77a25f40fb584a913d"
I20260809 06:47:16.075109 12335 raft_consensus.cc:3037] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:47:16.075165 11930 raft_consensus.cc:3037] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:47:16.077322 12335 raft_consensus.cc:2454] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c29759cc124693b9dd46151ada086e in term 1.
I20260809 06:47:16.077373 11930 raft_consensus.cc:2454] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c29759cc124693b9dd46151ada086e in term 1.
I20260809 06:47:16.077733 12032 leader_election.cc:302] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98082a206958458c8cdec165cdc3893c, f4c29759cc124693b9dd46151ada086e; no voters:
I20260809 06:47:16.078037 12444 raft_consensus.cc:2781] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:47:16.078471 12444 raft_consensus.cc:686] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 1 LEADER]: Becoming Leader. State: Replica: f4c29759cc124693b9dd46151ada086e, State: Running, Role: LEADER
I20260809 06:47:16.078930 12444 consensus_queue.cc:237] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } }
I20260809 06:47:16.083837 11716 catalog_manager.cc:5495] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c29759cc124693b9dd46151ada086e (127.27.209.194). New cstate: current_term: 1 leader_uuid: "f4c29759cc124693b9dd46151ada086e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c29759cc124693b9dd46151ada086e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 45757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 } health_report { overall_health: UNKNOWN } } }
I20260809 06:47:16.108723 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.103637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.27.209.194:45757)
0809 06:47:16.103857 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:16.103863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:16.103883 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:47:16.103977 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:16.103981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:16.104064 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:16.104190 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:47:16.104586 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.105330 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.106618 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.107026 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.107141 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.107631 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:47:16.108138 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:47:16.108270 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:16.108385 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 06:47:16.108468 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:47:16.116770 12335 raft_consensus.cc:1261] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 1 FOLLOWER]: Refusing update from remote peer f4c29759cc124693b9dd46151ada086e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:47:16.116786 11930 raft_consensus.cc:1261] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 1 FOLLOWER]: Refusing update from remote peer f4c29759cc124693b9dd46151ada086e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:47:16.117687 12448 consensus_queue.cc:1035] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "98082a206958458c8cdec165cdc3893c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:47:16.118011 12444 consensus_queue.cc:1035] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdddf394ed094c77a25f40fb584a913d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 42687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:47:16.128134 12451 mvcc.cc:205] Tried to move back new op lower bound from 7316512915924746240 to 7316512915783331840. Current Snapshot: MvccSnapshot[applied={T|T < 7316512915924746240}]
I20260809 06:47:16.144176 11716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:47:16.144423 11716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:47:16.147816 11716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 001b124d91f3453fa5b1cc805c823782: Sending DeleteTablet for 3 replicas of tablet 5d9816eee2fc4dcfa5dade6edc56271d
I20260809 06:47:16.149025 12306 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9816eee2fc4dcfa5dade6edc56271d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:47:16 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:51782
I20260809 06:47:16.149132 12103 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9816eee2fc4dcfa5dade6edc56271d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:47:16 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:60508
I20260809 06:47:16.149127 11903 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9816eee2fc4dcfa5dade6edc56271d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:47:16 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:52486
I20260809 06:47:16.151490 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 11810
I20260809 06:47:16.160053 12472 tablet_replica.cc:330] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: stopping tablet replica
I20260809 06:47:16.160681 12471 tablet_replica.cc:330] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: stopping tablet replica
I20260809 06:47:16.161350 12471 raft_consensus.cc:2229] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:47:16.161868 12472 raft_consensus.cc:2229] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:47:16.162515 12472 raft_consensus.cc:2258] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:47:16.163225 12471 raft_consensus.cc:2258] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:47:16.173072 12469 tablet_replica.cc:330] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: stopping tablet replica
I20260809 06:47:16.173522 12469 raft_consensus.cc:2229] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:47:16.174052 12469 raft_consensus.cc:2258] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:47:16.176216 12472 ts_tablet_manager.cc:1905] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:47:16 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:47:16.177842 12471 ts_tablet_manager.cc:1905] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:47:16.179214 12469 ts_tablet_manager.cc:1905] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:47:16.184846 12472 ts_tablet_manager.cc:1918] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:47:16.184911 12471 ts_tablet_manager.cc:1918] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:47:16.185115 12471 log.cc:1196] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/5d9816eee2fc4dcfa5dade6edc56271d
I20260809 06:47:16.185245 12472 log.cc:1196] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/5d9816eee2fc4dcfa5dade6edc56271d
I20260809 06:47:16.185882 12472 ts_tablet_manager.cc:1939] T 5d9816eee2fc4dcfa5dade6edc56271d P fdddf394ed094c77a25f40fb584a913d: Deleting consensus metadata
I20260809 06:47:16.189100 11697 catalog_manager.cc:4841] TS fdddf394ed094c77a25f40fb584a913d (127.27.209.193:42687): tablet 5d9816eee2fc4dcfa5dade6edc56271d (table test-table [id=339b86d1f7ff49e989d6834f8fc6a9e8]) successfully deleted
I20260809 06:47:16.191608 12469 ts_tablet_manager.cc:1918] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:47:16.191782 12469 log.cc:1196] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/5d9816eee2fc4dcfa5dade6edc56271d
I20260809 06:47:16.192261 12469 ts_tablet_manager.cc:1939] T 5d9816eee2fc4dcfa5dade6edc56271d P 98082a206958458c8cdec165cdc3893c: Deleting consensus metadata
I20260809 06:47:16.192725 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.152641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.27.209.193:42687)
0809 06:47:16.152846 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:16.152854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:16.166150 (+ 13296us) server_negotiation.cc:407] Connection header received
0809 06:47:16.166424 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:16.166430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:16.166506 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:16.166577 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:16.168204 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.169294 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.171092 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.171748 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.171837 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.184794 (+ 12957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:16.184811 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:16.184835 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:16.184891 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:16.188668 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:16.189759 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:16.189770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:16.189774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:16.189948 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:16.190462 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:16.190468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:16.190471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:16.190932 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:16.191233 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:16.192042 (+ 809us) server_negotiation.cc:299] Negotiation successful
0809 06:47:16.192281 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:47:16.194247 11699 catalog_manager.cc:4841] TS 98082a206958458c8cdec165cdc3893c (127.27.209.195:33671): tablet 5d9816eee2fc4dcfa5dade6edc56271d (table test-table [id=339b86d1f7ff49e989d6834f8fc6a9e8]) successfully deleted
I20260809 06:47:16.194693 11991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:16.195631 12471 ts_tablet_manager.cc:1939] T 5d9816eee2fc4dcfa5dade6edc56271d P f4c29759cc124693b9dd46151ada086e: Deleting consensus metadata
I20260809 06:47:16.198873 11698 catalog_manager.cc:4841] TS f4c29759cc124693b9dd46151ada086e (127.27.209.194:45757): tablet 5d9816eee2fc4dcfa5dade6edc56271d (table test-table [id=339b86d1f7ff49e989d6834f8fc6a9e8]) successfully deleted
I20260809 06:47:16.292622 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 11810
I20260809 06:47:16.322901 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 12008
Aug 09 06:47:16 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:47:16.339836 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.323667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.27.209.194:45757)
0809 06:47:16.323832 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:16.323836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:16.323898 (+ 62us) server_negotiation.cc:407] Connection header received
0809 06:47:16.324077 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:16.324080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:16.324141 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:16.324237 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:16.325858 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.326708 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.328642 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.329073 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.329172 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.333975 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:16.333986 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:16.334005 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:16.334043 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:16.337035 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:16.337671 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:16.337683 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:16.337686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:16.337832 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:16.338221 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:16.338225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:16.338227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:16.338587 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:16.338910 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:16.339216 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 06:47:16.339437 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:47:16.340448 12190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:16.454358 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 12008
I20260809 06:47:16.478417 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 12207
Aug 09 06:47:16 dist-test-slave-jpj1 krb5kdc[11653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:47:16.492113 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:47:16.479163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.27.209.195:33671)
0809 06:47:16.479354 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 06:47:16.479358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:47:16.479378 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:47:16.479528 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:47:16.479561 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:47:16.479635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:47:16.479718 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:47:16.480888 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.481591 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.482752 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:47:16.483115 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:47:16.483209 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:47:16.487278 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:47:16.487292 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:47:16.487310 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:47:16.487349 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:47:16.489796 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:16.490362 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:16.490368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:16.490371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:16.490499 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:47:16.490779 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:47:16.490781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:47:16.490783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:47:16.491117 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:47:16.491342 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 06:47:16.491520 (+ 178us) server_negotiation.cc:299] Negotiation successful
0809 06:47:16.491829 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:47:16.492796 12399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:16.580222 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 12207
I20260809 06:47:16.604029 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 11672
I20260809 06:47:16.605206 11757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:47:16.713310 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 11672
2026-08-09T06:47:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4625 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:47:16 dist-test-slave-jpj1 krb5kdc[12483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:47:16 dist-test-slave-jpj1 krb5kdc[12483](info): set up 2 sockets
Aug 09 06:47:16 dist-test-slave-jpj1 krb5kdc[12483](info): commencing operation
krb5kdc: starting...
W20260809 06:47:17.939924 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:47:18 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:47:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/rangeradmin_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/rangeradmin_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/rangerlookup_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/rangerlookup_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
I20260809 06:47:18.298060 28487 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHh9PjI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/postgres -l logfile start
2026-08-09 06:47:20.623 UTC [12538] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 06:47:20.623 UTC [12538] LOG: listening on IPv4 address "127.27.209.212", port 47323
2026-08-09 06:47:20.627 UTC [12538] LOG: listening on Unix socket "/tmp/.s.PGSQL.47323"
2026-08-09 06:47:20.648 UTC [12543] LOG: database system was shut down at 2026-08-09 06:47:19 UTC
2026-08-09 06:47:20.652 UTC [12538] LOG: database system is ready to accept connections
I20260809 06:47:21.723272 28487 mini_postgres.cc:96] Postgres bound to 47323
2026-08-09 06:47:21.745 UTC [12554] FATAL: database "slave" does not exist
127.27.209.212:47323 - accepting connections
I20260809 06:47:21.746783 28487 mini_ranger.cc:161] Starting Ranger...
I20260809 06:47:21.775972 28487 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 06:47:22.199333 28487 mini_ranger.cc:87] Created ranger Postgres database
I20260809 06:47:22.199478 28487 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-admin
2026-08-09 06:47:22,534 [I] DB FLAVOR :POSTGRES
2026-08-09 06:47:22,535 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:47:22,535 [I] Checking connection..
2026-08-09 06:47:22,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 06:47:22,794 [I] Checking connection passed.
2026-08-09 06:47:22,794 [I] --------- Verifying version history table ---------
2026-08-09 06:47:22,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:47:23,042 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:47:23,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:47:23,282 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:47:23,282 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 06:47:23,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 06:47:23.525 UTC [12643] WARNING: there is no transaction in progress
2026-08-09 06:47:23,531 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 06:47:23,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:47:23,786 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 06:47:23,787 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:47:23,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 06:47:24,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 06:47:24,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jpj1.c.gcp-upstream.internal','N') ;"
2026-08-09 06:47:24,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:47:24,777 [I] Table x_portal_user does not exist in database ranger
2026-08-09 06:47:24,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:47:25,033 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 06:47:25,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:47:25,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:47:25,514 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 06:47:25,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 06:47:26.426 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.441 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.455 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.470 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.484 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.499 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.549 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.555 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.568 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.577 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.587 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.598 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.607 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.613 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.618 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:26.624 UTC [12831] WARNING: there is no transaction in progress
2026-08-09 06:47:27,205 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 06:47:27,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 06:47:27,462 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 06:47:27,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:47:27,709 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 06:47:27,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:47:27,944 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 06:47:27,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:47:28,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:47:28,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jpj1.c.gcp-upstream.internal';"
2026-08-09 06:47:28,688 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 06:47:28,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 06:47:28,924 [I] DB_PATCHES have already been applied
I20260809 06:47:28.930799 28487 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHh9PjI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 06:47:28.930886 28487 mini_ranger.cc:193] Using host: 127.27.209.212
I20260809 06:47:28.949870 28487 mini_ranger.cc:227] Ranger admin URL: http://127.27.209.212:38953
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 6:47:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 06:47:30.190903 28487 mini_ranger.cc:160] Time spent starting Ranger: real 8.444s user 0.000s sys 0.054s
Aug 09, 2026 6:47:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 6:47:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 6:47:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 6:47:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 6:47:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 06:47:50.397823 28487 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/rangerkms_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/rangerkms_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.209.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.209.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 06:47:50 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258070, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 06:47:50.588869 28487 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 06:47:50.612602 28487 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 06:47:51.347177 28487 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 06:47:51.347355 28487 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms
I20260809 06:47:51.347479 28487 mini_ranger_kms.cc:227] Using postgres at 127.27.209.212:47323
2026-08-09 06:47:52,156 [I] DB FLAVOR :POSTGRES
2026-08-09 06:47:52,156 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:47:52,157 [I] Checking connection
2026-08-09 06:47:52,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 06:47:52,389 [I] connection success
2026-08-09 06:47:52,389 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 06:47:52,389 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 06:47:52,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 06:47:52,650 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 06:47:52,650 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:47:52,650 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 06:47:52,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:47323/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 06:47:52,907 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 06:47:53.367128 28487 mini_ranger_kms.cc:310] Created kms service
I20260809 06:47:53.563890 28487 mini_ranger_kms.cc:326] Created kudu user
I20260809 06:47:53.604183 28487 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 06:47:53 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258070, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:47:53.956117 28487 mini_ranger_kms.cc:384] Added ranger policy
I20260809 06:47:53.957430 28487 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 06:47:53.957517 28487 mini_ranger_kms.cc:240] Using host: 127.27.209.212
I20260809 06:47:53.987358 28487 mini_ranger_kms.cc:276] Ranger KMS PID: 13129
I20260809 06:47:53.987519 28487 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.209.212:56429
Aug 09, 2026 6:47:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:47:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:47:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 6:47:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 6:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 6:47:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 06:47:56 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786258076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.209.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:47:56.578651 28487 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.990s user 0.139s sys 0.386s
I20260809 06:47:56.579028 28487 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 06:47:56.579272 28487 mini_ranger_kms.cc:401] 127.27.209.212:56429/kms/v1/keys
Aug 09 06:47:56 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.209.212@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
Aug 09 06:47:57 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258076, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.209.212@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
Aug 09 06:47:59 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:47:59.429201 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:34835
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:34835
--ranger_config_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:56429/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:47:59.915058 13211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:47:59.915458 13211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:47:59.915640 13211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:47:59.924782 13211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:47:59.924983 13211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:47:59.925132 13211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:47:59.925288 13211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:47:59.925426 13211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:47:59.937812 13211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.209.212:56429/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:34835
--ranger_config_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:34835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:47:59.939177 13211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:47:59.940667 13211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:47:59.956063 13228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:59.956215 13224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:47:59.956367 13226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:47:59.957911 13211 server_base.cc:1034] running on GCE node
I20260809 06:47:59.958813 13211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:47:59.960542 13211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:47:59.961805 13211 hybrid_clock.cc:648] HybridClock initialized: now 1786258079961728 us; error 31 us; skew 500 ppm
I20260809 06:47:59.962234 13211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:47:59 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:47:59.966437 13211 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:47:59.967962 13211 webserver.cc:460] Webserver started at http://127.27.209.254:43199/ using document root <none> and password file <none>
I20260809 06:47:59.968576 13211 fs_manager.cc:359] Metadata directory not provided
I20260809 06:47:59.968734 13211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:47:59.969105 13211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:47:59 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:00.025650 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "d381e648351c47e2b3604410a4fba384"
format_stamp: "Formatted at 2026-08-09 06:48:00 on dist-test-slave-jpj1"
server_key: "d10663a7b84c6fca30c3d05791080f21"
server_key_iv: "5ff82b796ed738095701ef98b23fb492"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:00.026616 13211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "d381e648351c47e2b3604410a4fba384"
format_stamp: "Formatted at 2026-08-09 06:48:00 on dist-test-slave-jpj1"
server_key: "d10663a7b84c6fca30c3d05791080f21"
server_key_iv: "5ff82b796ed738095701ef98b23fb492"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:00.033334 13211 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 06:48:00.052367 13237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:00.054025 13211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 06:48:00.054281 13211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "d381e648351c47e2b3604410a4fba384"
format_stamp: "Formatted at 2026-08-09 06:48:00 on dist-test-slave-jpj1"
server_key: "d10663a7b84c6fca30c3d05791080f21"
server_key_iv: "5ff82b796ed738095701ef98b23fb492"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:00.054540 13211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:00.086021 13211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:00.091100 13211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:00.091394 13211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:00.127731 13211 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:34835
I20260809 06:48:00.127920 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:34835 every 8 connection(s)
I20260809 06:48:00.129432 13211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:48:00.131735 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 13211
I20260809 06:48:00.131944 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:48:00.132701 28487 external_mini_cluster.cc:1428] Setting key fb2c498d926645e01ae9fa7dbb22250b
I20260809 06:48:00.135525 13327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:48:00 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:00.153477 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384: Bootstrap starting.
I20260809 06:48:00.155709 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:00.134969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.27.209.254:34835)
0809 06:48:00.135955 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:00.135971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:00.136014 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:48:00.137651 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:00.137672 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:00.138050 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:00.138409 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:00.139770 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:00.141107 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:00.142591 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:00.143442 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:00.143657 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:00.147389 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:00.147401 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:00.147421 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:00.147492 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:00.151679 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:00.152346 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:00.152354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:00.152360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:00.152504 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:00.152890 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:00.152894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:00.152896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:00.153477 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:00.153805 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:00.154217 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 06:48:00.154641 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":551,"threads_started":1}
I20260809 06:48:00.158516 13327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:00.160439 13327 log.cc:824] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:00.163977 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384: No bootstrap required, opened a new log
I20260809 06:48:00.179345 13327 raft_consensus.cc:348] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381e648351c47e2b3604410a4fba384" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34835 } }
I20260809 06:48:00.180063 13327 raft_consensus.cc:374] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:00.180181 13327 raft_consensus.cc:729] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d381e648351c47e2b3604410a4fba384, State: Initialized, Role: FOLLOWER
I20260809 06:48:00.180899 13327 consensus_queue.cc:260] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381e648351c47e2b3604410a4fba384" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34835 } }
I20260809 06:48:00.181185 13327 raft_consensus.cc:388] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:48:00.181348 13327 raft_consensus.cc:482] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:48:00.181633 13327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:00.185297 13327 raft_consensus.cc:504] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381e648351c47e2b3604410a4fba384" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34835 } }
I20260809 06:48:00.186000 13327 leader_election.cc:302] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d381e648351c47e2b3604410a4fba384; no voters:
I20260809 06:48:00.186945 13327 leader_election.cc:288] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:48:00.186952 13341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:48:00.188059 13341 raft_consensus.cc:686] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [term 1 LEADER]: Becoming Leader. State: Replica: d381e648351c47e2b3604410a4fba384, State: Running, Role: LEADER
I20260809 06:48:00.188710 13341 consensus_queue.cc:237] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381e648351c47e2b3604410a4fba384" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34835 } }
I20260809 06:48:00.189713 13327 sys_catalog.cc:564] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:48:00.193783 13345 sys_catalog.cc:455] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d381e648351c47e2b3604410a4fba384. Latest consensus state: current_term: 1 leader_uuid: "d381e648351c47e2b3604410a4fba384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381e648351c47e2b3604410a4fba384" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34835 } } }
I20260809 06:48:00.194222 13345 sys_catalog.cc:458] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:00.195181 13344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d381e648351c47e2b3604410a4fba384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381e648351c47e2b3604410a4fba384" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 34835 } } }
I20260809 06:48:00.195900 13344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:00.201951 13352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:48:00.218351 13352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:48:00.231828 13352 catalog_manager.cc:1347] Generated new cluster ID: 0657c2fc4b724746a5828774cec325d0
I20260809 06:48:00.232096 13352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:48:00.235080 13327 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 06:48:00.254498 13352 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:48:00.256974 13352 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:48:00.267495 13352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384: Generated new TSK 0
I20260809 06:48:00.268534 13352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:48:01.104230 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:56429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:34835
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:48:01 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786258081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 06:48:01.634328 13412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:01.634682 13412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:01.634920 13412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:01.644703 13412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:01.644959 13412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:01.645161 13412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:48:01.657140 13412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.209.212:56429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:34835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:01.658595 13412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:01.660220 13412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:01.677878 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:01.682351 13429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:01.685952 13412 server_base.cc:1034] running on GCE node
W20260809 06:48:01.686305 13432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:01.686988 13412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:01.688088 13412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:01.689396 13412 hybrid_clock.cc:648] HybridClock initialized: now 1786258081689319 us; error 56 us; skew 500 ppm
I20260809 06:48:01.690115 13412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:01 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:01.694952 13412 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:48:01.705361 13412 webserver.cc:460] Webserver started at http://127.27.209.193:44117/ using document root <none> and password file <none>
I20260809 06:48:01.706192 13412 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:01.706365 13412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:01.706832 13412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:01 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:01.731274 13412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "0a285f496eb949e1b9f22c858c912e63"
format_stamp: "Formatted at 2026-08-09 06:48:01 on dist-test-slave-jpj1"
server_key: "a1c5821f7a3a0f7d13d95ca658b05b4b"
server_key_iv: "837872bf4dd5963e0a53363dafe8d30f"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:01.732509 13412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "0a285f496eb949e1b9f22c858c912e63"
format_stamp: "Formatted at 2026-08-09 06:48:01 on dist-test-slave-jpj1"
server_key: "a1c5821f7a3a0f7d13d95ca658b05b4b"
server_key_iv: "837872bf4dd5963e0a53363dafe8d30f"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:01.741019 13412 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260809 06:48:01.762791 13443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:01.764979 13412 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260809 06:48:01.765265 13412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "0a285f496eb949e1b9f22c858c912e63"
format_stamp: "Formatted at 2026-08-09 06:48:01 on dist-test-slave-jpj1"
server_key: "a1c5821f7a3a0f7d13d95ca658b05b4b"
server_key_iv: "837872bf4dd5963e0a53363dafe8d30f"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:01.765769 13412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:01.821759 13412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:01.828008 13412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:01.828729 13412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:01.833276 13412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:01.835225 13412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:01.835326 13412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:01.835507 13412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:01.835736 13412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:48:01 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:02.082756 13412 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:35151
I20260809 06:48:02.082772 13617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:35151 every 8 connection(s)
I20260809 06:48:02.084504 13412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:48:02.090857 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 13412
I20260809 06:48:02.091095 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:48:02.091967 28487 external_mini_cluster.cc:1428] Setting key 8befa8355010255739f3768c729a7161
Aug 09 06:48:02 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:02.119835 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:02.093662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:34835 (local address 127.27.209.193:38773)
0809 06:48:02.094537 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:02.094594 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:02.097439 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:02.098650 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:02.098665 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:02.099436 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:02.100115 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:02.100135 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.101202 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.101206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:02.102498 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:02.102511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.103315 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.103322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:02.103577 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:02.108822 (+ 5245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:02.108848 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:02.111000 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:02.114862 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:02.114872 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:02.114891 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:02.115524 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:02.116037 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:02.116042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:02.116046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:02.116320 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:02.117053 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:02.117062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:02.117526 (+ 464us) client_negotiation.cc:769] Sending connection context
0809 06:48:02.117850 (+ 324us) client_negotiation.cc:240] Negotiation successful
0809 06:48:02.118399 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":510,"threads_started":1}
I20260809 06:48:02.121981 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:02.093040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:38773 (local address 127.27.209.254:34835)
0809 06:48:02.093602 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:02.093609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:02.094713 (+ 1104us) server_negotiation.cc:407] Connection header received
0809 06:48:02.097825 (+ 3112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:02.097832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:02.097955 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:02.098041 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:02.100297 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.101043 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.102647 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.103085 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.103208 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:02.111255 (+ 8047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:02.111260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:02.111264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:02.111300 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:02.114671 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:02.115721 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:02.115727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:02.115731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:02.115897 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:02.116458 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:02.116464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:02.116467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:02.116838 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:02.116978 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:02.121340 (+ 4362us) server_negotiation.cc:299] Negotiation successful
0809 06:48:02.121579 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260809 06:48:02.123919 13619 heartbeater.cc:347] Connected to a master server at 127.27.209.254:34835
I20260809 06:48:02.124235 13619 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:02.125145 13619 heartbeater.cc:511] Master 127.27.209.254:34835 requested a full tablet report, sending...
I20260809 06:48:02.127815 13270 ts_manager.cc:194] Registered new tserver with Master: 0a285f496eb949e1b9f22c858c912e63 (127.27.209.193:35151)
I20260809 06:48:02.129886 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:38773
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:48:02.175231 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:56429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:34835
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:48:02 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
W20260809 06:48:02.676565 13628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:02.676870 13628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:02.677006 13628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:02.685184 13628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:02.685315 13628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:02.685463 13628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:48:02.695497 13628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.209.212:56429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:34835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:02.696852 13628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:02.698168 13628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:02.712934 13646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:02.713155 13650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:02.712898 13647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:02.715008 13628 server_base.cc:1034] running on GCE node
I20260809 06:48:02.715725 13628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:02.716809 13628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:02.718042 13628 hybrid_clock.cc:648] HybridClock initialized: now 1786258082717988 us; error 37 us; skew 500 ppm
I20260809 06:48:02.718400 13628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:02 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:02.722189 13628 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:48:02.723830 13628 webserver.cc:460] Webserver started at http://127.27.209.194:37713/ using document root <none> and password file <none>
I20260809 06:48:02.724674 13628 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:02.724805 13628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:02.725142 13628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:02 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:02.743691 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "4a38e8ca07264126b720a7f98b989b13"
format_stamp: "Formatted at 2026-08-09 06:48:02 on dist-test-slave-jpj1"
server_key: "59cd1fa28d550aa699b3162b2f04fa52"
server_key_iv: "0319c39521539a80aa19db63f356496f"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:02.744762 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "4a38e8ca07264126b720a7f98b989b13"
format_stamp: "Formatted at 2026-08-09 06:48:02 on dist-test-slave-jpj1"
server_key: "59cd1fa28d550aa699b3162b2f04fa52"
server_key_iv: "0319c39521539a80aa19db63f356496f"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:02.751813 13628 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260809 06:48:02.770701 13660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:02.772629 13628 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260809 06:48:02.772835 13628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "4a38e8ca07264126b720a7f98b989b13"
format_stamp: "Formatted at 2026-08-09 06:48:02 on dist-test-slave-jpj1"
server_key: "59cd1fa28d550aa699b3162b2f04fa52"
server_key_iv: "0319c39521539a80aa19db63f356496f"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:02.773082 13628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:02.798775 13628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:02.804427 13628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:02.804759 13628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:02.806051 13628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:02.807809 13628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:02.807902 13628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:02.808025 13628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:02.808084 13628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:02.876861 13628 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:41857
I20260809 06:48:02.876925 13837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:41857 every 8 connection(s)
I20260809 06:48:02.878548 13628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:48:02.883106 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 13628
I20260809 06:48:02.883307 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:48:02.883973 28487 external_mini_cluster.cc:1428] Setting key 73e73588a77f208cb3993c01052ed078
Aug 09 06:48:02 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
I20260809 06:48:02.923667 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:02.882872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:36193 (local address 127.27.209.254:34835)
0809 06:48:02.903962 (+ 21090us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:02.903967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:02.903991 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:48:02.904055 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:02.904059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:02.904140 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:02.904268 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:02.906706 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.907853 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.910971 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.911609 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.911747 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:02.916112 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:02.916120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:02.916125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:02.916174 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:02.919758 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:02.920770 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:02.920775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:02.920777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:02.920913 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:02.921442 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:02.921445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:02.921447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:02.921784 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:02.921896 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:02.922976 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0809 06:48:02.923225 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20956,"thread_start_us":20838,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
I20260809 06:48:02.924817 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:02.882240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:34835 (local address 127.27.209.194:36193)
0809 06:48:02.888098 (+ 5858us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:02.888165 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:02.890416 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:02.904719 (+ 14303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:02.904734 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:02.905439 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:02.906430 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:02.906460 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.908097 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.908101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:02.910784 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:02.910802 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:02.911856 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:02.911863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:02.912156 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:02.913000 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:02.913041 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:02.915810 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:02.919949 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:02.919960 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:02.919981 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:02.920622 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:02.921025 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:02.921030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:02.921033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:02.921338 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:02.921937 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:02.921946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:02.922420 (+ 474us) client_negotiation.cc:769] Sending connection context
0809 06:48:02.922756 (+ 336us) client_negotiation.cc:240] Negotiation successful
0809 06:48:02.923333 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5547,"thread_start_us":5331,"threads_started":1}
I20260809 06:48:02.926508 13840 heartbeater.cc:347] Connected to a master server at 127.27.209.254:34835
I20260809 06:48:02.926867 13840 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:02.927718 13840 heartbeater.cc:511] Master 127.27.209.254:34835 requested a full tablet report, sending...
I20260809 06:48:02.929560 13270 ts_manager.cc:194] Registered new tserver with Master: 4a38e8ca07264126b720a7f98b989b13 (127.27.209.194:41857)
I20260809 06:48:02.930696 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:36193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:48:02.954792 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:56429/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:34835
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:48:03.133957 13619 heartbeater.cc:503] Master 127.27.209.254:34835 was elected leader, sending a full tablet report...
W20260809 06:48:03.333420 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:03.333725 13848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:03.333838 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:03.341193 13848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:03.341306 13848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:03.341427 13848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:48:03.351135 13848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.209.212:56429/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:34835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:03.352517 13848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:03.353884 13848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:03.366590 13857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:03.366676 13860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:03.366706 13858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:03.367697 13848 server_base.cc:1034] running on GCE node
I20260809 06:48:03.368257 13848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:03.369145 13848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:03.370340 13848 hybrid_clock.cc:648] HybridClock initialized: now 1786258083370282 us; error 52 us; skew 500 ppm
I20260809 06:48:03.370633 13848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:03 dist-test-slave-jpj1 krb5kdc[12483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:03.374056 13848 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:48:03.375365 13848 webserver.cc:460] Webserver started at http://127.27.209.195:43091/ using document root <none> and password file <none>
I20260809 06:48:03.376036 13848 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:03.376138 13848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:03.376389 13848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:03 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:03.392903 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "8dce67b334bc4512877e07f8f729f0ca"
format_stamp: "Formatted at 2026-08-09 06:48:03 on dist-test-slave-jpj1"
server_key: "23b06f96417698199c68db8834e892cf"
server_key_iv: "18b18eb96b885881bedacc8af9f27b54"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:03.394024 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "8dce67b334bc4512877e07f8f729f0ca"
format_stamp: "Formatted at 2026-08-09 06:48:03 on dist-test-slave-jpj1"
server_key: "23b06f96417698199c68db8834e892cf"
server_key_iv: "18b18eb96b885881bedacc8af9f27b54"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:03.401669 13848 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260809 06:48:03.420653 13869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:03.422159 13848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:48:03.422308 13848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8dce67b334bc4512877e07f8f729f0ca"
format_stamp: "Formatted at 2026-08-09 06:48:03 on dist-test-slave-jpj1"
server_key: "23b06f96417698199c68db8834e892cf"
server_key_iv: "18b18eb96b885881bedacc8af9f27b54"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:03.422518 13848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:03.443851 13848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:03.447747 13848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:03.448014 13848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:03.449137 13848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:03.450518 13848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:03.450608 13848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:03.450794 13848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:03.450891 13848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:03.486683 13848 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:37145
I20260809 06:48:03.486704 14052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:37145 every 8 connection(s)
I20260809 06:48:03.487998 13848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:48:03.491503 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 13848
I20260809 06:48:03.491685 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:48:03.492098 28487 external_mini_cluster.cc:1428] Setting key 099a45bc6b5cb233b642f1a21ec2b8e5
Aug 09 06:48:03 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258083, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:03.508126 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.490472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:60841 (local address 127.27.209.254:34835)
0809 06:48:03.490931 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.490936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.492079 (+ 1143us) server_negotiation.cc:407] Connection header received
0809 06:48:03.494150 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.494156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.494235 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.494322 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:03.495960 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.496697 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.498247 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.498668 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.498779 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.502473 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:03.502478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:03.502481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:03.502513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:03.505076 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.505799 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.505805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.505809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.505973 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.506441 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.506446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.506449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.506795 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:03.506959 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.507696 (+ 737us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.507872 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260809 06:48:03.509055 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.491064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:34835 (local address 127.27.209.195:60841)
0809 06:48:03.491916 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:03.491962 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:03.493891 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:03.494580 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:03.494592 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:03.495201 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:03.495822 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.495843 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.496847 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.496851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.498127 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.498139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.498951 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.498958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.499346 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.500124 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:03.500146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:03.502280 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:03.505209 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:03.505215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:03.505227 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:03.505658 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:03.506075 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:03.506080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:03.506083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:03.506324 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:03.506921 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:03.506928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:03.507323 (+ 395us) client_negotiation.cc:769] Sending connection context
0809 06:48:03.507600 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 06:48:03.507976 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":379,"threads_started":1}
I20260809 06:48:03.510242 14055 heartbeater.cc:347] Connected to a master server at 127.27.209.254:34835
I20260809 06:48:03.510520 14055 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:03.511273 14055 heartbeater.cc:511] Master 127.27.209.254:34835 requested a full tablet report, sending...
I20260809 06:48:03.512631 13270 ts_manager.cc:194] Registered new tserver with Master: 8dce67b334bc4512877e07f8f729f0ca (127.27.209.195:37145)
I20260809 06:48:03.513445 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:60841
I20260809 06:48:03.519290 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:48:03.535718 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.523716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.27.209.254:34835)
0809 06:48:03.523886 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.523893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.524229 (+ 336us) server_negotiation.cc:407] Connection header received
0809 06:48:03.524400 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.524404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.524473 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.524554 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:03.525971 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.526783 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.528186 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.528576 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.528684 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.530897 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:03.530902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:03.530905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:03.530935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:03.533218 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.533861 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.533866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.533870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.533989 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.534323 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.534328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.534332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.534595 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:03.534831 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.535170 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.535398 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:48:03.539983 13270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:48:03.547437 13270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:48:03.582754 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.560810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.27.209.194:41857)
0809 06:48:03.561835 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.561846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.573352 (+ 11506us) server_negotiation.cc:407] Connection header received
0809 06:48:03.573838 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.573849 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.574148 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.574323 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:03.575337 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.576912 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.579923 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.581304 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.581474 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.581655 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.582287 (+ 632us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.582425 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":654,"threads_started":1}
I20260809 06:48:03.583604 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.560330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37145 (local address 127.0.0.1:49644)
0809 06:48:03.562416 (+ 2086us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:03.562463 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:03.562625 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:03.576663 (+ 14038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:03.576668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:03.576695 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:03.577069 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.577079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.578693 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.578698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.580864 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.580881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.582656 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.582662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.582829 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.582859 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:48:03.582947 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 06:48:03.583141 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"thread_start_us":1704,"threads_started":1}
I20260809 06:48:03.583946 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.560602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:41857 (local address 127.0.0.1:42594)
0809 06:48:03.573213 (+ 12611us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:03.573232 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:03.573613 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:03.574668 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:03.574676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:03.574710 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:03.575140 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.575152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.577226 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.577230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.579785 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.579801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.581812 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.581818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.582006 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.582065 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 06:48:03.583040 (+ 975us) client_negotiation.cc:240] Negotiation successful
0809 06:48:03.583156 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12464,"mutex_wait_us":2,"thread_start_us":12896,"threads_started":1}
I20260809 06:48:03.584347 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.560622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.27.209.195:37145)
0809 06:48:03.576109 (+ 15487us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.576117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.576148 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:48:03.576231 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.576250 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.576470 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.576643 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:03.577254 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.578565 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.581024 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.582443 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.582730 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.583154 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.583222 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.583381 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15317,"thread_start_us":16238,"threads_started":1}
I20260809 06:48:03.586094 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.562826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:35151 (local address 127.0.0.1:56418)
0809 06:48:03.573270 (+ 10444us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:03.573290 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:03.573421 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:03.579089 (+ 5668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:03.579093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:03.579119 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:03.579652 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.579669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.581077 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.581082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.583072 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.583088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.585427 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.585434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.585548 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.585570 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:48:03.585648 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:48:03.585772 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10306,"thread_start_us":10559,"threads_started":1}
I20260809 06:48:03.586557 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.563208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.27.209.193:35151)
0809 06:48:03.578542 (+ 15334us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.578549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.578577 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:48:03.578675 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.578683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.578889 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.579080 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:03.579870 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.580911 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.583268 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.584952 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.585122 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.585288 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.585720 (+ 432us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.585880 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15175,"thread_start_us":14882,"threads_started":1}
I20260809 06:48:03.587954 13950 tablet_service.cc:1467] Processing CreateTablet for tablet 86ed6f04cc2142cdbe08807dfcef1da4 (DEFAULT_TABLE table=test-table [id=b3651321b6554f8fb4e706e412227955]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:48:03.588053 13734 tablet_service.cc:1467] Processing CreateTablet for tablet 86ed6f04cc2142cdbe08807dfcef1da4 (DEFAULT_TABLE table=test-table [id=b3651321b6554f8fb4e706e412227955]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:48:03.589676 13950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ed6f04cc2142cdbe08807dfcef1da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:03.589738 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ed6f04cc2142cdbe08807dfcef1da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:03.589828 13515 tablet_service.cc:1467] Processing CreateTablet for tablet 86ed6f04cc2142cdbe08807dfcef1da4 (DEFAULT_TABLE table=test-table [id=b3651321b6554f8fb4e706e412227955]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:48:03.591053 13515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ed6f04cc2142cdbe08807dfcef1da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:03.602340 14088 tablet_bootstrap.cc:492] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Bootstrap starting.
I20260809 06:48:03.605108 14090 tablet_bootstrap.cc:492] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Bootstrap starting.
I20260809 06:48:03.606612 14088 tablet_bootstrap.cc:654] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:03.607969 14088 log.cc:824] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:03.608372 14089 tablet_bootstrap.cc:492] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Bootstrap starting.
I20260809 06:48:03.609138 14090 tablet_bootstrap.cc:654] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:03.610451 14090 log.cc:824] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:03.611856 14089 tablet_bootstrap.cc:654] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:03.613083 14089 log.cc:824] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:03.613642 14090 tablet_bootstrap.cc:492] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: No bootstrap required, opened a new log
I20260809 06:48:03.613916 14090 ts_tablet_manager.cc:1397] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 06:48:03.614128 14088 tablet_bootstrap.cc:492] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: No bootstrap required, opened a new log
I20260809 06:48:03.614377 14088 ts_tablet_manager.cc:1397] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260809 06:48:03.616622 14089 tablet_bootstrap.cc:492] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: No bootstrap required, opened a new log
I20260809 06:48:03.616868 14089 ts_tablet_manager.cc:1397] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260809 06:48:03.622248 14090 raft_consensus.cc:348] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.622586 14090 raft_consensus.cc:374] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:03.622677 14090 raft_consensus.cc:729] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dce67b334bc4512877e07f8f729f0ca, State: Initialized, Role: FOLLOWER
I20260809 06:48:03.623227 14090 consensus_queue.cc:260] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.624215 14055 heartbeater.cc:503] Master 127.27.209.254:34835 was elected leader, sending a full tablet report...
I20260809 06:48:03.624979 14088 raft_consensus.cc:348] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.625378 14088 raft_consensus.cc:374] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:03.625473 14088 raft_consensus.cc:729] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a285f496eb949e1b9f22c858c912e63, State: Initialized, Role: FOLLOWER
I20260809 06:48:03.626130 14088 consensus_queue.cc:260] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.626394 14090 ts_tablet_manager.cc:1428] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Time spent starting tablet: real 0.012s user 0.003s sys 0.008s
I20260809 06:48:03.627197 14089 raft_consensus.cc:348] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.627611 14089 raft_consensus.cc:374] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:03.627712 14089 raft_consensus.cc:729] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a38e8ca07264126b720a7f98b989b13, State: Initialized, Role: FOLLOWER
I20260809 06:48:03.628118 14088 ts_tablet_manager.cc:1428] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260809 06:48:03.628384 14089 consensus_queue.cc:260] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.629814 13840 heartbeater.cc:503] Master 127.27.209.254:34835 was elected leader, sending a full tablet report...
I20260809 06:48:03.631587 14089 ts_tablet_manager.cc:1428] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260809 06:48:03.636659 13842 tablet.cc:2369] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:48:03.722674 14098 raft_consensus.cc:482] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:48:03.722975 14098 raft_consensus.cc:504] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.725005 14098 leader_election.cc:288] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a38e8ca07264126b720a7f98b989b13 (127.27.209.194:41857), 0a285f496eb949e1b9f22c858c912e63 (127.27.209.193:35151)
I20260809 06:48:03.731271 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.725220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:35151 (local address 127.27.209.195:52923)
0809 06:48:03.725408 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:03.725424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:03.725630 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:03.726000 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:03.726005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:03.726031 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:03.726311 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.726322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.727513 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.727519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.729362 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.729376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.730560 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.730566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.730768 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.730805 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:48:03.730911 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 06:48:03.731052 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 06:48:03.731369 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.725476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:52923 (local address 127.27.209.193:35151)
0809 06:48:03.725698 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.725704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.725724 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:48:03.725796 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.725801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.725874 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.725943 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:03.726465 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.727388 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.729531 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.730391 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.730539 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.730597 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.730926 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.731020 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:48:03.732623 13545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ed6f04cc2142cdbe08807dfcef1da4" candidate_uuid: "8dce67b334bc4512877e07f8f729f0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a285f496eb949e1b9f22c858c912e63" is_pre_election: true
I20260809 06:48:03.732914 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.725978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:41857 (local address 127.27.209.195:48155)
0809 06:48:03.727039 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:03.727059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:03.727196 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:03.727573 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:03.727577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:03.727603 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:03.728145 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.728159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.729326 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.729331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.731004 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:03.731017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.732383 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.732387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:03.732502 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.732526 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:48:03.732604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:48:03.732688 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":716,"threads_started":1}
I20260809 06:48:03.733018 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.725641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:48155 (local address 127.27.209.194:41857)
0809 06:48:03.725912 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.725918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.727161 (+ 1243us) server_negotiation.cc:407] Connection header received
0809 06:48:03.727362 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.727367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.727446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.727552 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:03.728303 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.729163 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.731172 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.732193 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.732361 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.732429 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.732649 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.732775 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:48:03.733263 13545 raft_consensus.cc:2454] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dce67b334bc4512877e07f8f729f0ca in term 0.
I20260809 06:48:03.733943 13872 leader_election.cc:302] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a285f496eb949e1b9f22c858c912e63, 8dce67b334bc4512877e07f8f729f0ca; no voters:
I20260809 06:48:03.733961 13760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ed6f04cc2142cdbe08807dfcef1da4" candidate_uuid: "8dce67b334bc4512877e07f8f729f0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38e8ca07264126b720a7f98b989b13" is_pre_election: true
I20260809 06:48:03.734449 14098 raft_consensus.cc:2781] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:48:03.734500 13760 raft_consensus.cc:2454] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dce67b334bc4512877e07f8f729f0ca in term 0.
I20260809 06:48:03.734580 14098 raft_consensus.cc:482] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:48:03.734675 14098 raft_consensus.cc:3037] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:03.737155 14098 raft_consensus.cc:504] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.737768 14098 leader_election.cc:288] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [CANDIDATE]: Term 1 election: Requested vote from peers 4a38e8ca07264126b720a7f98b989b13 (127.27.209.194:41857), 0a285f496eb949e1b9f22c858c912e63 (127.27.209.193:35151)
I20260809 06:48:03.738206 13760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ed6f04cc2142cdbe08807dfcef1da4" candidate_uuid: "8dce67b334bc4512877e07f8f729f0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a38e8ca07264126b720a7f98b989b13"
I20260809 06:48:03.738364 13545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ed6f04cc2142cdbe08807dfcef1da4" candidate_uuid: "8dce67b334bc4512877e07f8f729f0ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a285f496eb949e1b9f22c858c912e63"
I20260809 06:48:03.738415 13760 raft_consensus.cc:3037] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:03.738610 13545 raft_consensus.cc:3037] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 0 FOLLOWER]: Advancing to term 1
W20260809 06:48:03.739502 14056 tablet.cc:2369] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:48:03.740913 13545 raft_consensus.cc:2454] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dce67b334bc4512877e07f8f729f0ca in term 1.
I20260809 06:48:03.741071 13760 raft_consensus.cc:2454] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dce67b334bc4512877e07f8f729f0ca in term 1.
I20260809 06:48:03.741274 13872 leader_election.cc:302] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a285f496eb949e1b9f22c858c912e63, 8dce67b334bc4512877e07f8f729f0ca; no voters:
I20260809 06:48:03.741567 14098 raft_consensus.cc:2781] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:48:03.741958 14098 raft_consensus.cc:686] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 1 LEADER]: Becoming Leader. State: Replica: 8dce67b334bc4512877e07f8f729f0ca, State: Running, Role: LEADER
I20260809 06:48:03.742463 14098 consensus_queue.cc:237] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } }
I20260809 06:48:03.746825 13270 catalog_manager.cc:5495] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dce67b334bc4512877e07f8f729f0ca (127.27.209.195). New cstate: current_term: 1 leader_uuid: "8dce67b334bc4512877e07f8f729f0ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dce67b334bc4512877e07f8f729f0ca" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 } health_report { overall_health: UNKNOWN } } }
I20260809 06:48:03.771874 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.766441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.27.209.195:37145)
0809 06:48:03.766613 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.766618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.766707 (+ 89us) server_negotiation.cc:407] Connection header received
0809 06:48:03.766851 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.766855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.766932 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.767050 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:48:03.767344 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.768255 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.769637 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.770138 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.770236 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.770839 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:48:03.771328 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:48:03.771437 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.771481 (+ 44us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.771577 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:48:03.780646 13545 raft_consensus.cc:1261] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 1 FOLLOWER]: Refusing update from remote peer 8dce67b334bc4512877e07f8f729f0ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:48:03.780782 13760 raft_consensus.cc:1261] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 1 FOLLOWER]: Refusing update from remote peer 8dce67b334bc4512877e07f8f729f0ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:48:03.781611 14104 consensus_queue.cc:1035] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a285f496eb949e1b9f22c858c912e63" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:48:03.782008 14098 consensus_queue.cc:1035] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a38e8ca07264126b720a7f98b989b13" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:48:03.797374 14112 mvcc.cc:205] Tried to move back new op lower bound from 7316513111154692096 to 7316513111012683776. Current Snapshot: MvccSnapshot[applied={T|T < 7316513111154692096}]
I20260809 06:48:03.798581 14111 mvcc.cc:205] Tried to move back new op lower bound from 7316513111154692096 to 7316513111012683776. Current Snapshot: MvccSnapshot[applied={T|T < 7316513111154692096}]
I20260809 06:48:03.805184 13269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:48:03.805503 13269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:48:03.809233 13269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d381e648351c47e2b3604410a4fba384: Sending DeleteTablet for 3 replicas of tablet 86ed6f04cc2142cdbe08807dfcef1da4
I20260809 06:48:03.810359 13734 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ed6f04cc2142cdbe08807dfcef1da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:48:03 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42594
I20260809 06:48:03.810420 13515 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ed6f04cc2142cdbe08807dfcef1da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:48:03 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56418
I20260809 06:48:03.811074 13950 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ed6f04cc2142cdbe08807dfcef1da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:48:03 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:49644
I20260809 06:48:03.811697 14120 tablet_replica.cc:330] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: stopping tablet replica
I20260809 06:48:03.812188 14120 raft_consensus.cc:2229] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:48:03.812733 14120 raft_consensus.cc:2258] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:48:03.812759 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 13412
I20260809 06:48:03.813566 14119 tablet_replica.cc:330] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: stopping tablet replica
I20260809 06:48:03.814060 14119 raft_consensus.cc:2229] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:48:03.814610 14119 raft_consensus.cc:2258] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:48:03.815397 14120 ts_tablet_manager.cc:1905] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:48:03.816927 14119 ts_tablet_manager.cc:1905] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:48:03.822835 14120 ts_tablet_manager.cc:1918] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:48:03.823006 14120 log.cc:1196] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/86ed6f04cc2142cdbe08807dfcef1da4
I20260809 06:48:03.823572 14120 ts_tablet_manager.cc:1939] T 86ed6f04cc2142cdbe08807dfcef1da4 P 0a285f496eb949e1b9f22c858c912e63: Deleting consensus metadata
I20260809 06:48:03.823935 14119 ts_tablet_manager.cc:1918] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:48:03.824150 14119 log.cc:1196] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/86ed6f04cc2142cdbe08807dfcef1da4
I20260809 06:48:03.824257 14123 tablet_replica.cc:330] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: stopping tablet replica
I20260809 06:48:03.824728 14119 ts_tablet_manager.cc:1939] T 86ed6f04cc2142cdbe08807dfcef1da4 P 4a38e8ca07264126b720a7f98b989b13: Deleting consensus metadata
I20260809 06:48:03.824676 14123 raft_consensus.cc:2229] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:48:03.825477 13243 catalog_manager.cc:4841] TS 0a285f496eb949e1b9f22c858c912e63 (127.27.209.193:35151): tablet 86ed6f04cc2142cdbe08807dfcef1da4 (table test-table [id=b3651321b6554f8fb4e706e412227955]) successfully deleted
I20260809 06:48:03.826004 14123 raft_consensus.cc:2258] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:48:03.826443 13241 catalog_manager.cc:4841] TS 4a38e8ca07264126b720a7f98b989b13 (127.27.209.194:41857): tablet 86ed6f04cc2142cdbe08807dfcef1da4 (table test-table [id=b3651321b6554f8fb4e706e412227955]) successfully deleted
I20260809 06:48:03.828163 14123 ts_tablet_manager.cc:1905] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:48:03 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:48:03.835670 14123 ts_tablet_manager.cc:1918] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:48:03.835835 14123 log.cc:1196] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/86ed6f04cc2142cdbe08807dfcef1da4
I20260809 06:48:03.836383 14123 ts_tablet_manager.cc:1939] T 86ed6f04cc2142cdbe08807dfcef1da4 P 8dce67b334bc4512877e07f8f729f0ca: Deleting consensus metadata
I20260809 06:48:03.838100 13243 catalog_manager.cc:4841] TS 8dce67b334bc4512877e07f8f729f0ca (127.27.209.195:37145): tablet 86ed6f04cc2142cdbe08807dfcef1da4 (table test-table [id=b3651321b6554f8fb4e706e412227955]) successfully deleted
I20260809 06:48:03.839287 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.815441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.27.209.193:35151)
0809 06:48:03.815655 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.815661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.824663 (+ 9002us) server_negotiation.cc:407] Connection header received
0809 06:48:03.825110 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.825115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.825195 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.825273 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:03.826893 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.827733 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.829167 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.829594 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.829704 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.833714 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:03.833727 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:03.833743 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:03.833773 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:03.836598 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.837281 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.837287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.837290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.837414 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.837764 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.837769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.837772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.838230 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:03.838479 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.838813 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.838988 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:48:03.839901 13609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:03.944875 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 13412
I20260809 06:48:03.976601 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 13628
Aug 09 06:48:03 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:48:03.993131 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:03.977378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.27.209.194:41857)
0809 06:48:03.977573 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:03.977578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:03.977600 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:48:03.977713 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:03.977719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:03.977826 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:03.977937 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:03.979147 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.980381 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.981613 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:03.982236 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:03.982369 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:03.986372 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:03.986390 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:03.986411 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:03.986466 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:03.990313 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.991057 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.991066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.991070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.991241 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:03.991595 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:03.991600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:03.991604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:03.992072 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:03.992368 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:03.992547 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 06:48:03.992760 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:48:03.993870 13825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:04.082882 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 13628
I20260809 06:48:04.106907 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 13848
Aug 09 06:48:04 dist-test-slave-jpj1 krb5kdc[12483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:48:04.125128 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:04.107701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.27.209.195:37145)
0809 06:48:04.107870 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:04.107874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:04.107971 (+ 97us) server_negotiation.cc:407] Connection header received
0809 06:48:04.108165 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:04.108169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:04.108236 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:04.108319 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:04.110237 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:04.111084 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:04.113115 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:04.113590 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:04.113699 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:04.118901 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:04.118915 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:04.118932 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:04.118986 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:04.122302 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:04.123028 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:04.123036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:04.123038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:04.123196 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:04.123586 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:04.123590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:04.123593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:04.123958 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:04.124297 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:04.124555 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 06:48:04.124784 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:48:04.125847 14048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:04.251761 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 13848
I20260809 06:48:04.274612 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 13211
I20260809 06:48:04.275919 13320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:04.387401 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 13211
I20260809 06:48:04.413805 28487 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 06:48:04.758204 28487 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 06:48:04.758504 28487 mini_ranger.cc:66] Stopping Ranger...
I20260809 06:48:05.102813 28487 mini_ranger.cc:68] Stopped Ranger
2026-08-09 06:48:05.103 UTC [12538] LOG: received smart shutdown request
2026-08-09 06:48:05.106 UTC [12538] LOG: background worker "logical replication launcher" (PID 12551) exited with exit code 1
2026-08-09 06:48:05.107 UTC [12546] LOG: shutting down
2026-08-09 06:48:05.251 UTC [12538] LOG: database system is shut down
2026-08-09T06:48:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48637 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:48:05 dist-test-slave-jpj1 krb5kdc[14144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:48:05 dist-test-slave-jpj1 krb5kdc[14144](info): set up 2 sockets
Aug 09 06:48:05 dist-test-slave-jpj1 krb5kdc[14144](info): commencing operation
krb5kdc: starting...
W20260809 06:48:06.564713 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:48:06 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:48:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/rangeradmin_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/rangeradmin_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/rangerlookup_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/rangerlookup_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
I20260809 06:48:06.898689 28487 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskHh9PjI/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/postgres -l logfile start
2026-08-09 06:48:09.149 UTC [14196] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 06:48:09.149 UTC [14196] LOG: listening on IPv4 address "127.27.209.212", port 42159
2026-08-09 06:48:09.152 UTC [14196] LOG: listening on Unix socket "/tmp/.s.PGSQL.42159"
2026-08-09 06:48:09.173 UTC [14200] LOG: database system was shut down at 2026-08-09 06:48:07 UTC
2026-08-09 06:48:09.176 UTC [14196] LOG: database system is ready to accept connections
I20260809 06:48:10.262881 28487 mini_postgres.cc:96] Postgres bound to 42159
2026-08-09 06:48:10.283 UTC [14213] FATAL: database "slave" does not exist
127.27.209.212:42159 - accepting connections
I20260809 06:48:10.284400 28487 mini_ranger.cc:161] Starting Ranger...
I20260809 06:48:10.311118 28487 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 06:48:10.706038 28487 mini_ranger.cc:87] Created ranger Postgres database
I20260809 06:48:10.706159 28487 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-admin
2026-08-09 06:48:10,988 [I] DB FLAVOR :POSTGRES
2026-08-09 06:48:10,989 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:48:10,989 [I] Checking connection..
2026-08-09 06:48:10,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 06:48:11,250 [I] Checking connection passed.
2026-08-09 06:48:11,250 [I] --------- Verifying version history table ---------
2026-08-09 06:48:11,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:48:11,508 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:48:11,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:48:11,763 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 06:48:11,763 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 06:48:11,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 06:48:12.010 UTC [14316] WARNING: there is no transaction in progress
2026-08-09 06:48:12,017 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 06:48:12,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 06:48:12,262 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 06:48:12,262 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:48:12,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 06:48:12,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 06:48:12,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jpj1.c.gcp-upstream.internal','N') ;"
2026-08-09 06:48:12,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:48:13,239 [I] Table x_portal_user does not exist in database ranger
2026-08-09 06:48:13,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:48:13,491 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 06:48:13,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:48:13,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:48:13,974 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 06:48:13,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 06:48:14.870 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.884 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.898 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.912 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.926 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.943 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.993 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:14.998 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.011 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.020 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.030 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.043 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.052 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.057 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.063 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15.069 UTC [14483] WARNING: there is no transaction in progress
2026-08-09 06:48:15,631 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 06:48:15,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 06:48:15,869 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 06:48:15,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 06:48:16,115 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 06:48:16,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 06:48:16,374 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 06:48:16,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 06:48:16,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 06:48:16,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jpj1.c.gcp-upstream.internal';"
2026-08-09 06:48:17,083 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 06:48:17,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 06:48:17,344 [I] DB_PATCHES have already been applied
I20260809 06:48:17.350835 28487 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-admin:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskHh9PjI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 06:48:17.350929 28487 mini_ranger.cc:193] Using host: 127.27.209.212
I20260809 06:48:17.368886 28487 mini_ranger.cc:227] Ranger admin URL: http://127.27.209.212:51111
Aug 09, 2026 6:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 6:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 6:48:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 6:48:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 6:48:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 06:48:18.578114 28487 mini_ranger.cc:160] Time spent starting Ranger: real 8.294s user 0.004s sys 0.045s
Aug 09, 2026 6:48:38 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 6:48:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 09, 2026 6:48:38 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 6:48:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 6:48:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 06:48:40.001061 28487 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.209.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.209.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/rangerkms_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.209.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/rangerkms_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.209.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.209.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.209.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/HTTP_127.27.209.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 06:48:40 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258120, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 06:48:40.175784 28487 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 06:48:40.197460 28487 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 06:48:40.910424 28487 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 06:48:40.910530 28487 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms
I20260809 06:48:40.910573 28487 mini_ranger_kms.cc:227] Using postgres at 127.27.209.212:42159
2026-08-09 06:48:41,271 [I] DB FLAVOR :POSTGRES
2026-08-09 06:48:41,271 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 06:48:41,272 [I] Checking connection
2026-08-09 06:48:41,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 06:48:41,506 [I] connection success
2026-08-09 06:48:41,506 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 06:48:41,506 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 06:48:41,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 06:48:41,741 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 06:48:41,742 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 06:48:41,742 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 06:48:41,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.209.212:42159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 06:48:41,988 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 06:48:42.434127 28487 mini_ranger_kms.cc:310] Created kms service
I20260809 06:48:42.643213 28487 mini_ranger_kms.cc:326] Created kudu user
I20260809 06:48:42.684691 28487 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 06:48:42 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258120, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:43.035693 28487 mini_ranger_kms.cc:384] Added ranger policy
I20260809 06:48:43.036073 28487 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms:/tmp/dist-test-taskHh9PjI/build/asan/bin/postgresql.jar:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskHh9PjI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 06:48:43.036155 28487 mini_ranger_kms.cc:240] Using host: 127.27.209.212
I20260809 06:48:43.057014 28487 mini_ranger_kms.cc:276] Ranger KMS PID: 14773
I20260809 06:48:43.057150 28487 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.209.212:54609
Aug 09, 2026 6:48:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 6:48:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 6:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 6:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 6:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 6:48:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 06:48:45 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786258125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.209.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:45.487068 28487 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.311s user 0.072s sys 0.316s
I20260809 06:48:45.487342 28487 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 06:48:45.487558 28487 mini_ranger_kms.cc:401] 127.27.209.212:54609/kms/v1/keys
Aug 09 06:48:45 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.209.212@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
Aug 09 06:48:45 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258125, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.209.212@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
Aug 09 06:48:48 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:48:48.278788 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37983
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:37983
--ranger_config_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:48:48.617900 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:48.618175 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:48.618270 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:48.625249 14846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:48:48.625344 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:48.625397 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:48.625452 14846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:48:48.625499 14846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:48:48.635066 14846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.209.212:54609/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:37983
--ranger_config_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:48.636226 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:48.637471 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:48.648579 14853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:48.648530 14855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:48.648617 14857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:48.649348 14846 server_base.cc:1034] running on GCE node
I20260809 06:48:48.650075 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:48.651367 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:48.652555 14846 hybrid_clock.cc:648] HybridClock initialized: now 1786258128652504 us; error 46 us; skew 500 ppm
I20260809 06:48:48.652900 14846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:48 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:48.655810 14846 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:48:48.657130 14846 webserver.cc:460] Webserver started at http://127.27.209.254:38285/ using document root <none> and password file <none>
I20260809 06:48:48.657738 14846 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:48.657847 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:48.658130 14846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:48 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:48.715232 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "893bfbfdaa0c4335850d0025ceb68e94"
format_stamp: "Formatted at 2026-08-09 06:48:48 on dist-test-slave-jpj1"
server_key: "7ca46a0ed5bf01e4bcfd4aa3fcc65fa4"
server_key_iv: "7a244f254b15bc68b586d042ddc4b26d"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:48.716130 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "893bfbfdaa0c4335850d0025ceb68e94"
format_stamp: "Formatted at 2026-08-09 06:48:48 on dist-test-slave-jpj1"
server_key: "7ca46a0ed5bf01e4bcfd4aa3fcc65fa4"
server_key_iv: "7a244f254b15bc68b586d042ddc4b26d"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:48.721447 14846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:48:48.737313 14865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:48.738919 14846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:48:48.739094 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "893bfbfdaa0c4335850d0025ceb68e94"
format_stamp: "Formatted at 2026-08-09 06:48:48 on dist-test-slave-jpj1"
server_key: "7ca46a0ed5bf01e4bcfd4aa3fcc65fa4"
server_key_iv: "7a244f254b15bc68b586d042ddc4b26d"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:48.739343 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:48.755594 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:48.759161 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:48.759446 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:48.776847 14846 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:37983
I20260809 06:48:48.776882 14946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:37983 every 8 connection(s)
I20260809 06:48:48.778087 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:48:48.782008 14947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:48.784938 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 14846
I20260809 06:48:48.785153 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:48:48.786000 28487 external_mini_cluster.cc:1428] Setting key 568e4024ff952bce96d76089d6ec758e
I20260809 06:48:48.795400 14947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Bootstrap starting.
I20260809 06:48:48.798624 14947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:48.799646 14947 log.cc:824] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Log is configured to *not* fsync() on all Append() calls
Aug 09 06:48:48 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:48.802964 14947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: No bootstrap required, opened a new log
I20260809 06:48:48.809791 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:48.788058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.27.209.254:37983)
0809 06:48:48.788862 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:48.788877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:48.788930 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:48:48.791068 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:48.791099 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:48.791644 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:48.792090 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:48.793210 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:48.795090 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:48.796430 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:48.797122 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:48.797301 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:48.801500 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:48.801512 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:48.801530 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:48.801571 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:48.805341 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:48.806001 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:48.806009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:48.806018 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:48.806224 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:48.806550 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:48.806555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:48.806558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:48.807269 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:48.807614 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:48.807953 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 06:48:48.808393 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":336,"threads_started":1}
I20260809 06:48:48.812672 14947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:48.813006 14947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:48.813081 14947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893bfbfdaa0c4335850d0025ceb68e94, State: Initialized, Role: FOLLOWER
I20260809 06:48:48.813619 14947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:48.813838 14947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:48:48.813941 14947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:48:48.814059 14947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:48.816113 14947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:48.816553 14947 leader_election.cc:302] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893bfbfdaa0c4335850d0025ceb68e94; no voters:
I20260809 06:48:48.817070 14947 leader_election.cc:288] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:48:48.817183 14955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:48:48.817826 14955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 LEADER]: Becoming Leader. State: Replica: 893bfbfdaa0c4335850d0025ceb68e94, State: Running, Role: LEADER
I20260809 06:48:48.818547 14955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:48.819233 14947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:48:48.821764 14957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "893bfbfdaa0c4335850d0025ceb68e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } } }
I20260809 06:48:48.822274 14957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:48.821229 14959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893bfbfdaa0c4335850d0025ceb68e94. Latest consensus state: current_term: 1 leader_uuid: "893bfbfdaa0c4335850d0025ceb68e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } } }
I20260809 06:48:48.822755 14959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:48.824442 14967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:48:48.829573 14967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:48:48.830185 14947 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 06:48:48.847366 14967 catalog_manager.cc:1347] Generated new cluster ID: bd55198993e64bec991dfa1203b1e4f3
I20260809 06:48:48.847571 14967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:48:48.894104 14967 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:48:48.896134 14967 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:48:48.908728 14967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Generated new TSK 0
I20260809 06:48:48.910012 14967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:48:49.679658 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37983
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:48:49 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786258129, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 06:48:50.138643 15014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:50.139146 15014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:50.139348 15014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:50.151024 15014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:50.151172 15014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:50.151368 15014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:48:50.163434 15014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.209.212:54609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:50.164938 15014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:50.166249 15014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:50.180447 15030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:50.181003 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:50.181360 15029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:50.183184 15014 server_base.cc:1034] running on GCE node
I20260809 06:48:50.184092 15014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:50.191609 15014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:50.192940 15014 hybrid_clock.cc:648] HybridClock initialized: now 1786258130192825 us; error 121 us; skew 500 ppm
I20260809 06:48:50.193364 15014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:50 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258130, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:50.202296 15014 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:48:50.204568 15014 webserver.cc:460] Webserver started at http://127.27.209.193:40477/ using document root <none> and password file <none>
I20260809 06:48:50.205583 15014 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:50.205742 15014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:50.206162 15014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:50 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258130, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:50.233788 15014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "11f87aa0f0394efdb90ca868a0a891fa"
format_stamp: "Formatted at 2026-08-09 06:48:50 on dist-test-slave-jpj1"
server_key: "cee61ed734fe0824866c8f925fc6d02f"
server_key_iv: "2a62be75deff96e81539285737ae20e1"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:50.235230 15014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "11f87aa0f0394efdb90ca868a0a891fa"
format_stamp: "Formatted at 2026-08-09 06:48:50 on dist-test-slave-jpj1"
server_key: "cee61ed734fe0824866c8f925fc6d02f"
server_key_iv: "2a62be75deff96e81539285737ae20e1"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:50.259651 15014 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.019s sys 0.004s
I20260809 06:48:50.286163 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:50.288177 15014 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260809 06:48:50.288546 15014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "11f87aa0f0394efdb90ca868a0a891fa"
format_stamp: "Formatted at 2026-08-09 06:48:50 on dist-test-slave-jpj1"
server_key: "cee61ed734fe0824866c8f925fc6d02f"
server_key_iv: "2a62be75deff96e81539285737ae20e1"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:50.288894 15014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:50.319700 15014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:50.325207 15014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:50.325592 15014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:50.327099 15014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:50.332490 15014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:50.332654 15014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 06:48:50.332847 15014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:50.332986 15014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:48:50 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258129, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:50.484609 15014 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:45239
I20260809 06:48:50.484680 15208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:45239 every 8 connection(s)
I20260809 06:48:50.486361 15014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:48:50.491802 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 15014
I20260809 06:48:50.492023 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:48:50.492867 28487 external_mini_cluster.cc:1428] Setting key e4cc34fd1ed4220eac46a5b875ecfa05
Aug 09 06:48:50 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258130, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
I20260809 06:48:50.535653 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:50.489574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:53519 (local address 127.27.209.254:37983)
0809 06:48:50.496544 (+ 6970us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:50.496552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:50.504995 (+ 8443us) server_negotiation.cc:407] Connection header received
0809 06:48:50.507860 (+ 2865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:50.507867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:50.507961 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:50.508045 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:50.510380 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:50.511518 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:50.513877 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:50.514540 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:50.514692 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:50.527710 (+ 13018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:50.527716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:50.527719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:50.527768 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:50.531308 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:50.532450 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:50.532457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:50.532461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:50.532649 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:50.533264 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:50.533269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:50.533272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:50.533654 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:50.533802 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:50.534955 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0809 06:48:50.535201 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6799,"thread_start_us":6618,"threads_started":1}
I20260809 06:48:50.538378 15214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:50.490136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37983 (local address 127.27.209.193:53519)
0809 06:48:50.504805 (+ 14669us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:50.504860 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:50.507318 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:50.508661 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:50.508684 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:50.509284 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:50.510177 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:50.510204 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:50.511758 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:50.511762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:50.513651 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:50.513666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:50.514809 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:50.514815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:50.515094 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:50.518589 (+ 3495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:50.518623 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:50.521664 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:50.531563 (+ 9899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:50.531575 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:50.531598 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:50.532284 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:50.532825 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:50.532829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:50.532833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:50.533119 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:50.533912 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:50.533921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:50.534402 (+ 481us) client_negotiation.cc:769] Sending connection context
0809 06:48:50.536233 (+ 1831us) client_negotiation.cc:240] Negotiation successful
0809 06:48:50.536847 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14343,"thread_start_us":14133,"threads_started":1}
I20260809 06:48:50.542128 15210 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37983
I20260809 06:48:50.542495 15210 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:50.543435 15210 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:50.545976 14896 ts_manager.cc:194] Registered new tserver with Master: 11f87aa0f0394efdb90ca868a0a891fa (127.27.209.193:45239)
I20260809 06:48:50.547991 14896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:53519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:48:50.562505 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37983
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:48:50 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258129, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
W20260809 06:48:50.969307 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:50.969753 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:50.969956 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:50.978020 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:50.978188 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:50.978389 15221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:48:50.992949 15221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.209.212:54609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:50.995164 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:50.996968 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:51.013902 15238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:51.013931 15237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:51.014700 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:51.015113 15221 server_base.cc:1034] running on GCE node
I20260809 06:48:51.015935 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:51.017230 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:51.018486 15221 hybrid_clock.cc:648] HybridClock initialized: now 1786258131018416 us; error 52 us; skew 500 ppm
I20260809 06:48:51.018895 15221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:51.023104 15221 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:48:51.024974 15221 webserver.cc:460] Webserver started at http://127.27.209.194:36069/ using document root <none> and password file <none>
I20260809 06:48:51.025888 15221 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:51.026041 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:51.026428 15221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:51.044593 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "62974c8a94e141bb96e9b74737fd791e"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "788eefcd6b5d39408373d3feb4c32f53"
server_key_iv: "8cb2c3bf4c9f80e3abfdc01cb48f934c"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:51.045475 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "62974c8a94e141bb96e9b74737fd791e"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "788eefcd6b5d39408373d3feb4c32f53"
server_key_iv: "8cb2c3bf4c9f80e3abfdc01cb48f934c"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:51.051348 15221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:48:51.068939 15254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:51.070822 15221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 06:48:51.071005 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "62974c8a94e141bb96e9b74737fd791e"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "788eefcd6b5d39408373d3feb4c32f53"
server_key_iv: "8cb2c3bf4c9f80e3abfdc01cb48f934c"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:51.071264 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:51.095320 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:51.100514 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:51.100847 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:51.102010 15221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:51.103818 15221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:51.103904 15221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:51.104032 15221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:51.104087 15221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:51.140744 15221 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:37727
I20260809 06:48:51.140820 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:37727 every 8 connection(s)
I20260809 06:48:51.142169 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:48:51.149287 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 15221
I20260809 06:48:51.149461 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:48:51.150009 28487 external_mini_cluster.cc:1428] Setting key 52a4c5e74177136aa959f9d49ee90579
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:51.163322 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.145126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:55125 (local address 127.27.209.254:37983)
0809 06:48:51.145683 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:51.145690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:51.146744 (+ 1054us) server_negotiation.cc:407] Connection header received
0809 06:48:51.149342 (+ 2598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:51.149348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:51.149452 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:51.149577 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:51.151440 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.152360 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.153931 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.154334 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.154447 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.157671 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:51.157677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:51.157680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:51.157711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:51.160311 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.161049 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.161053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.161055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.161177 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.161609 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.161614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.161617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.161889 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:51.162022 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:51.162710 (+ 688us) server_negotiation.cc:299] Negotiation successful
0809 06:48:51.162948 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
I20260809 06:48:51.164361 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.145636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37983 (local address 127.27.209.194:55125)
0809 06:48:51.146473 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:51.146535 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:51.149016 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:51.149870 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:51.149883 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:51.150459 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:51.151249 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:51.151275 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.152533 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.152537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:51.153789 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:51.153803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.154553 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.154561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:51.154777 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.155398 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:51.155420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:51.157464 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:51.160457 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:51.160467 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:51.160485 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:51.160937 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:51.161284 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:51.161289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:51.161293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:51.161500 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:51.162015 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:51.162023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:51.162349 (+ 326us) client_negotiation.cc:769] Sending connection context
0809 06:48:51.162592 (+ 243us) client_negotiation.cc:240] Negotiation successful
0809 06:48:51.163115 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":309,"threads_started":1}
I20260809 06:48:51.166028 15428 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37983
I20260809 06:48:51.166368 15428 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:51.167219 15428 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:51.168958 14896 ts_manager.cc:194] Registered new tserver with Master: 62974c8a94e141bb96e9b74737fd791e (127.27.209.194:37727)
I20260809 06:48:51.170025 14896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:55125
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:48:51.212219 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37983
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:48:51.540266 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:51.540524 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:51.540652 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:51.547729 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:51.547827 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:51.547974 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:48:51.551604 15210 heartbeater.cc:503] Master 127.27.209.254:37983 was elected leader, sending a full tablet report...
I20260809 06:48:51.557895 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.209.212:54609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:51.559139 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:51.560418 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:51.572926 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:51.572991 15444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:51.573047 15442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:51.574287 15434 server_base.cc:1034] running on GCE node
I20260809 06:48:51.574882 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:51.575867 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:51.577107 15434 hybrid_clock.cc:648] HybridClock initialized: now 1786258131577031 us; error 50 us; skew 500 ppm
I20260809 06:48:51.577414 15434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:51.580471 15434 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:48:51.581840 15434 webserver.cc:460] Webserver started at http://127.27.209.195:37747/ using document root <none> and password file <none>
I20260809 06:48:51.582507 15434 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:51.582626 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:51.582923 15434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:51.597846 15434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "8b8f683525014e59aca8f83b007727d4"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "103c4d93dce91293e344a29735cec698"
server_key_iv: "a64769e8c1302fbe1be5fed4304c02e8"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:51.598804 15434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "8b8f683525014e59aca8f83b007727d4"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "103c4d93dce91293e344a29735cec698"
server_key_iv: "a64769e8c1302fbe1be5fed4304c02e8"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:51.604312 15434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:48:51.620072 15455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:51.621626 15434 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 06:48:51.621789 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8b8f683525014e59aca8f83b007727d4"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "103c4d93dce91293e344a29735cec698"
server_key_iv: "a64769e8c1302fbe1be5fed4304c02e8"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:51.622004 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:51.654850 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:51.658675 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:51.658955 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:51.659906 15434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:51.661289 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:51.661375 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:51.661499 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:51.661557 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:51.692322 15434 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:45171
I20260809 06:48:51.692358 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:45171 every 8 connection(s)
I20260809 06:48:51.693562 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:48:51.695572 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 15434
I20260809 06:48:51.695773 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:48:51.696396 28487 external_mini_cluster.cc:1428] Setting key 3a1667b9f6c338b9c96e88bd1fe4ecb2
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:51.713344 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.696186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:48233 (local address 127.27.209.254:37983)
0809 06:48:51.696727 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:51.696732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:51.698151 (+ 1419us) server_negotiation.cc:407] Connection header received
0809 06:48:51.700200 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:51.700204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:51.700274 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:51.700362 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:51.701659 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.702355 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.703828 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.704234 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.704332 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.707345 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:51.707350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:51.707353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:51.707385 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:51.709659 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.710580 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.710585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.710588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.710910 (+ 322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.711488 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.711493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.711496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.712009 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:51.712163 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:51.712923 (+ 760us) server_negotiation.cc:299] Negotiation successful
0809 06:48:51.713096 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":278,"threads_started":1}
I20260809 06:48:51.714206 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.696890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37983 (local address 127.27.209.195:48233)
0809 06:48:51.697674 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:51.697721 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:51.699961 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:51.700551 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:51.700558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:51.700965 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:51.701533 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:51.701551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.702468 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.702471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:51.703724 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:51.703737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.704410 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.704415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:51.704607 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.705169 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:51.705190 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:51.707145 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:51.709784 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:51.709792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:51.709807 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:51.710400 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:51.711051 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:51.711056 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:51.711060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:51.711352 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:51.712157 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:51.712166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:51.712558 (+ 392us) client_negotiation.cc:769] Sending connection context
0809 06:48:51.712806 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 06:48:51.713210 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":313,"threads_started":1}
I20260809 06:48:51.715461 15624 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37983
I20260809 06:48:51.715780 15624 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:51.716454 15624 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:51.717794 14896 ts_manager.cc:194] Registered new tserver with Master: 8b8f683525014e59aca8f83b007727d4 (127.27.209.195:45171)
I20260809 06:48:51.718577 14896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:48233
I20260809 06:48:51.724054 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:48:51.725840 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 15014
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:48:51.741276 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.726782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.27.209.193:45239)
0809 06:48:51.727424 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:51.727430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:51.727453 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:48:51.727506 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:51.727512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:51.727747 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:51.727866 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:51.729233 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.730213 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.731508 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.732070 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.732163 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.736135 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:51.736146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:51.736168 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:51.736206 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:51.738937 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.739560 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.739569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.739572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.739727 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.740001 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.740004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.740006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.740340 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:51.740554 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:51.740786 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 06:48:51.740984 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":319,"threads_started":1}
I20260809 06:48:51.742565 15202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:51.825910 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 15014
I20260809 06:48:51.846949 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 15221
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:48:51.862538 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.847799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.27.209.194:37727)
0809 06:48:51.848363 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:51.848371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:51.848396 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:48:51.848460 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:51.848466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:51.848680 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:51.848810 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:51.850090 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.851087 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.852503 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.853058 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.853182 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.857193 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:51.857205 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:51.857220 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:51.857253 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:51.859815 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.860456 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.860463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.860465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.860594 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.860951 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.860954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.860956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.861328 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:51.861609 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:51.861943 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 06:48:51.862201 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":291,"threads_started":1}
I20260809 06:48:51.863867 15421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:51.945533 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 15221
I20260809 06:48:51.970837 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 15434
Aug 09 06:48:51 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:48:51.986577 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:51.971786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.27.209.195:45171)
0809 06:48:51.972459 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:51.972466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:51.972496 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:48:51.972557 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:51.972564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:51.972773 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:51.972901 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:51.974147 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.975135 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.976444 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:51.977001 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:51.977100 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:51.981260 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:51.981275 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:51.981292 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:51.981334 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:51.984189 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.984776 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.984782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.984784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.984915 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:51.985188 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:51.985190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:51.985192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:51.985534 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:51.985783 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:51.985994 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 06:48:51.986196 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
I20260809 06:48:51.987895 15616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:52.070914 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 15434
I20260809 06:48:52.094918 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 14846
I20260809 06:48:52.096344 14941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:52.202168 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 14846
I20260809 06:48:52.232023 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37983
--webserver_interface=127.27.209.254
--webserver_port=38285
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:37983
--ranger_config_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:48:52.558913 15642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:52.559230 15642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:52.559361 15642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:52.566720 15642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:48:52.566843 15642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:52.566911 15642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:52.566988 15642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:48:52.567054 15642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:48:52.577172 15642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:37983
--ranger_config_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=38285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:52.578474 15642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:52.579849 15642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:52.591442 15653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:52.591418 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:52.591475 15652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:52.592092 15642 server_base.cc:1034] running on GCE node
I20260809 06:48:52.592957 15642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:52.594267 15642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:52.595480 15642 hybrid_clock.cc:648] HybridClock initialized: now 1786258132595432 us; error 47 us; skew 500 ppm
I20260809 06:48:52.595855 15642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:52 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258132, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:52.598764 15642 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:48:52.600194 15642 webserver.cc:460] Webserver started at http://127.27.209.254:38285/ using document root <none> and password file <none>
I20260809 06:48:52.600806 15642 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:52.600916 15642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:48:52 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258132, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:52.645422 15642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:48:52.648543 15669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:52.650024 15642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:48:52.650203 15642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "893bfbfdaa0c4335850d0025ceb68e94"
format_stamp: "Formatted at 2026-08-09 06:48:48 on dist-test-slave-jpj1"
server_key: "7ca46a0ed5bf01e4bcfd4aa3fcc65fa4"
server_key_iv: "7a244f254b15bc68b586d042ddc4b26d"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:52.651152 15642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:52.671677 15642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:52.675230 15642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:52.675506 15642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:52.692751 15642 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:37983
I20260809 06:48:52.692763 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:37983 every 8 connection(s)
I20260809 06:48:52.694139 15642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:48:52.696302 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 15642
I20260809 06:48:52.697161 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:45239
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:37983
--webserver_port=40477
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:48:52.701319 15746 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:48:52.704758 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Bootstrap starting.
I20260809 06:48:52.724277 15746 log.cc:824] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:52.733809 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:48:52.734339 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Bootstrap complete.
I20260809 06:48:52.743230 15746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:52.744161 15746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 893bfbfdaa0c4335850d0025ceb68e94, State: Initialized, Role: FOLLOWER
I20260809 06:48:52.744753 15746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:52.744967 15746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:48:52.745074 15746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:48:52.745260 15746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:48:52.749060 15746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:52.749572 15746 leader_election.cc:302] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 893bfbfdaa0c4335850d0025ceb68e94; no voters:
I20260809 06:48:52.750191 15746 leader_election.cc:288] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:48:52.750327 15752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:48:52.751472 15752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [term 2 LEADER]: Becoming Leader. State: Replica: 893bfbfdaa0c4335850d0025ceb68e94, State: Running, Role: LEADER
I20260809 06:48:52.752106 15752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } }
I20260809 06:48:52.752444 15746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:48:52.754612 15754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 893bfbfdaa0c4335850d0025ceb68e94. Latest consensus state: current_term: 2 leader_uuid: "893bfbfdaa0c4335850d0025ceb68e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } } }
I20260809 06:48:52.754681 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "893bfbfdaa0c4335850d0025ceb68e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "893bfbfdaa0c4335850d0025ceb68e94" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37983 } } }
I20260809 06:48:52.754978 15754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:52.755054 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:52.762042 15762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:48:52.767889 15762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:48:52.768718 15746 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 06:48:52.769470 15762 catalog_manager.cc:1259] Loaded cluster ID: bd55198993e64bec991dfa1203b1e4f3
I20260809 06:48:52.769663 15762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:48:52.773607 15762 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:48:52.775074 15762 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Loaded TSK: 0
I20260809 06:48:52.776683 15762 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:48:53.146646 15748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:53.147136 15748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:53.147339 15748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:53.159193 15748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:53.159430 15748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:53.159724 15748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:48:53.175558 15748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:45239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=40477
--tserver_master_addrs=127.27.209.254:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:53.177676 15748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:53.179687 15748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:53.201139 15802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:53.201841 15804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:53.215791 15748 server_base.cc:1034] running on GCE node
W20260809 06:48:53.216735 15806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:53.217696 15748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:53.219031 15748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:53.220294 15748 hybrid_clock.cc:648] HybridClock initialized: now 1786258133220219 us; error 63 us; skew 500 ppm
I20260809 06:48:53.220716 15748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:53 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:53.225072 15748 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:48:53.239017 15748 webserver.cc:460] Webserver started at http://127.27.209.193:40477/ using document root <none> and password file <none>
I20260809 06:48:53.240322 15748 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:53.240583 15748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:48:53 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:53.269989 15748 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:48:53.275274 15814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:53.277325 15748 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260809 06:48:53.277603 15748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "11f87aa0f0394efdb90ca868a0a891fa"
format_stamp: "Formatted at 2026-08-09 06:48:50 on dist-test-slave-jpj1"
server_key: "cee61ed734fe0824866c8f925fc6d02f"
server_key_iv: "2a62be75deff96e81539285737ae20e1"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:53.279156 15748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:53.327878 15748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:53.332059 15748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:53.332338 15748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:53.334251 15748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:53.336094 15748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:53.336225 15748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:53.336340 15748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:53.336423 15748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:53.386798 15748 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:45239
I20260809 06:48:53.386853 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:45239 every 8 connection(s)
I20260809 06:48:53.388551 15748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:48:53.399170 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 15748
I20260809 06:48:53.400164 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:37727
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:37983
--webserver_port=36069
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:48:53 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:53.451453 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:53.392049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37983 (local address 127.27.209.193:40595)
0809 06:48:53.428100 (+ 36051us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:53.428199 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:53.430831 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:53.433795 (+ 2964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:53.433806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:53.434431 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:53.435185 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:53.435209 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:53.436902 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:53.436906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:53.438800 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:53.438826 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:53.440673 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:53.440683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:53.440921 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:53.441570 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:53.441595 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:53.443649 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:53.446910 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:53.446919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:53.446935 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:53.447389 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:53.447913 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:53.447918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:53.447922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:53.448136 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:53.448903 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:53.448910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:53.449250 (+ 340us) client_negotiation.cc:769] Sending connection context
0809 06:48:53.449490 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 06:48:53.450066 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34759,"thread_start_us":34326,"threads_started":1}
I20260809 06:48:53.451448 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:53.392032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:40595 (local address 127.27.209.254:37983)
0809 06:48:53.431359 (+ 39327us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:53.431383 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:53.431434 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:48:53.431955 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:53.431978 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:53.432534 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:53.432979 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:53.435422 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:53.436730 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:53.438991 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:53.440089 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:53.440273 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:53.443896 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:53.443903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:53.443918 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:53.443953 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:53.446764 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:53.447529 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:53.447560 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:53.447568 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:53.447790 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:53.448244 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:53.448249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:53.448253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:53.448757 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:53.448926 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:53.449772 (+ 846us) server_negotiation.cc:299] Negotiation successful
0809 06:48:53.450154 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34624,"thread_start_us":33810,"threads_started":1}
I20260809 06:48:53.455847 15980 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37983
I20260809 06:48:53.456158 15980 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:53.456873 15980 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
W20260809 06:48:53.459568 15980 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:37983 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:48:53.459791 15980 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:53.460033 15980 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:53.461211 15980 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:53.461469 15980 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
W20260809 06:48:53.462579 15980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:48:53.827087 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:53.827391 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:53.827587 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:53.834847 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:53.834964 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:53.835120 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:48:53.845832 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:37727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=36069
--tserver_master_addrs=127.27.209.254:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:53.847162 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:53.848490 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:53.862771 16009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:53.862778 16010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:53.863739 16014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:53.864297 15987 server_base.cc:1034] running on GCE node
I20260809 06:48:53.864912 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:53.865895 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:53.867092 15987 hybrid_clock.cc:648] HybridClock initialized: now 1786258133867033 us; error 54 us; skew 500 ppm
I20260809 06:48:53.867401 15987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:53 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:53.870752 15987 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:48:53.872684 15987 webserver.cc:460] Webserver started at http://127.27.209.194:36069/ using document root <none> and password file <none>
I20260809 06:48:53.873551 15987 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:53.873715 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:48:53 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:53.910471 15987 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.005s sys 0.007s
I20260809 06:48:53.920212 16025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:53.922235 15987 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.003s
I20260809 06:48:53.922443 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "62974c8a94e141bb96e9b74737fd791e"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "788eefcd6b5d39408373d3feb4c32f53"
server_key_iv: "8cb2c3bf4c9f80e3abfdc01cb48f934c"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:53.923920 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:53.960991 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:53.966393 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:53.966722 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:53.968003 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:53.969888 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:53.970013 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:53.970131 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:53.970220 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786258134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:54.061205 15987 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:37727
I20260809 06:48:54.061298 16194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:37727 every 8 connection(s)
I20260809 06:48:54.062810 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:48:54.070921 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 15987
I20260809 06:48:54.071671 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.209.212:54609/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:45171
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:37983
--webserver_port=37747
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:54.090004 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.066446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37983 (local address 127.27.209.194:38479)
0809 06:48:54.067855 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:54.067909 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:54.070349 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:54.071719 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:54.071732 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:54.072358 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:54.073203 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.073231 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.074668 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.074673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.076562 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.076578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.077683 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.077690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.077957 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.078677 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:54.078706 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:54.081420 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:54.085136 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:54.085146 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:54.085164 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:54.085616 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:54.086076 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:54.086081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:54.086084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:54.086353 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:54.087136 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:54.087145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:54.087621 (+ 476us) client_negotiation.cc:769] Sending connection context
0809 06:48:54.087951 (+ 330us) client_negotiation.cc:240] Negotiation successful
0809 06:48:54.088529 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":789,"threads_started":1}
I20260809 06:48:54.091393 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.065725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:38479 (local address 127.27.209.254:37983)
0809 06:48:54.071061 (+ 5336us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.071067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.071088 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:48:54.071153 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.071159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.071249 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.071367 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:54.073453 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.074486 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.076744 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.077419 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.077569 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.081712 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:54.081718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:54.081723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:54.081766 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:54.084965 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.085780 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.085786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.085789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.085938 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.086507 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.086513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.086516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.086901 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:54.087057 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.090788 (+ 3731us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.091039 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5202,"thread_start_us":5079,"threads_started":1}
I20260809 06:48:54.095000 16197 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37983
I20260809 06:48:54.095346 16197 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:54.096302 16197 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:54.098492 15695 ts_manager.cc:194] Registered new tserver with Master: 62974c8a94e141bb96e9b74737fd791e (127.27.209.194:37727)
I20260809 06:48:54.100739 15695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:38479
I20260809 06:48:54.462738 15980 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:54.463172 15980 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:54.464476 15695 ts_manager.cc:194] Registered new tserver with Master: 11f87aa0f0394efdb90ca868a0a891fa (127.27.209.193:45239)
I20260809 06:48:54.465642 15695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:40595
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
W20260809 06:48:54.567353 16208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:54.567822 16208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:54.567996 16208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:54.577064 16208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:54.577183 16208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:54.577337 16208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:48:54.587375 16208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.209.212:54609/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:45171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=37747
--tserver_master_addrs=127.27.209.254:37983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:54.588721 16208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:54.590080 16208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:54.605090 16217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:54.605926 16215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:54.606122 16208 server_base.cc:1034] running on GCE node
W20260809 06:48:54.606361 16220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:54.606999 16208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:54.608096 16208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:54.609316 16208 hybrid_clock.cc:648] HybridClock initialized: now 1786258134609247 us; error 55 us; skew 500 ppm
I20260809 06:48:54.609637 16208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:54.612893 16208 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:48:54.616744 16208 webserver.cc:460] Webserver started at http://127.27.209.195:37747/ using document root <none> and password file <none>
I20260809 06:48:54.617589 16208 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:54.617744 16208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:54.645846 16208 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.007s sys 0.003s
I20260809 06:48:54.648752 16233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:54.650159 16208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:48:54.650314 16208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8b8f683525014e59aca8f83b007727d4"
format_stamp: "Formatted at 2026-08-09 06:48:51 on dist-test-slave-jpj1"
server_key: "103c4d93dce91293e344a29735cec698"
server_key_iv: "a64769e8c1302fbe1be5fed4304c02e8"
server_key_version: "kuduclusterkey@0"
I20260809 06:48:54.651322 16208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:54.671555 16208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:54.675513 16208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:54.675818 16208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:54.676810 16208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:48:54.678229 16208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:48:54.678324 16208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:54.678439 16208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:48:54.678509 16208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:54.738000 16208 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:45171
I20260809 06:48:54.738046 16402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:45171 every 8 connection(s)
I20260809 06:48:54.739735 16208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:48:54.744750 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 16208
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:54.782948 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.743306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:37983 (local address 127.27.209.195:35813)
0809 06:48:54.752182 (+ 8876us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:54.752242 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:54.754710 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:54.758695 (+ 3985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:54.758708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:54.759416 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:54.760314 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.760340 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.762030 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.762036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.764208 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.764225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.768207 (+ 3982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.768215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.768486 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.770788 (+ 2302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:54.770819 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:54.773619 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:54.777639 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:54.777648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:54.777667 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:54.778243 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:54.778697 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:54.778701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:54.778705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:54.778942 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:54.780179 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:54.780188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:54.780603 (+ 415us) client_negotiation.cc:769] Sending connection context
0809 06:48:54.780942 (+ 339us) client_negotiation.cc:240] Negotiation successful
0809 06:48:54.781507 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8528,"thread_start_us":8279,"threads_started":1}
I20260809 06:48:54.784374 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.742692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:35813 (local address 127.27.209.254:37983)
0809 06:48:54.757900 (+ 15208us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.757905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.757926 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:48:54.757992 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.757997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.758116 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.758226 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:54.760543 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.761794 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.764398 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.765067 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.765213 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.773881 (+ 8668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:54.773893 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:54.773897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:54.773935 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:54.777436 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.778391 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.778397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.778400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.778556 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.779069 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.779074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.779077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.780004 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:54.780166 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.783740 (+ 3574us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.783973 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15096,"spinlock_wait_cycles":1408,"thread_start_us":14976,"threads_started":1}
I20260809 06:48:54.785760 16403 heartbeater.cc:347] Connected to a master server at 127.27.209.254:37983
I20260809 06:48:54.786096 16403 heartbeater.cc:464] Registering TS with master...
I20260809 06:48:54.786557 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.757865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.27.209.254:37983)
0809 06:48:54.765110 (+ 7245us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.765115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.765130 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:48:54.765173 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.765178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.765265 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.765379 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:54.767472 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.768580 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.774442 (+ 5862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.775140 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.775262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.779035 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:54.779042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:54.779046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:54.779079 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:54.782431 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.783274 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.783278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.783281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.783430 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.783931 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.783936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.783940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.784276 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:54.785836 (+ 1560us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.785970 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.786191 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7115,"thread_start_us":7678,"threads_started":1}
I20260809 06:48:54.787012 16403 heartbeater.cc:511] Master 127.27.209.254:37983 requested a full tablet report, sending...
I20260809 06:48:54.788944 15694 ts_manager.cc:194] Registered new tserver with Master: 8b8f683525014e59aca8f83b007727d4 (127.27.209.195:45171)
I20260809 06:48:54.790206 15694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:35813
Aug 09 06:48:54 dist-test-slave-jpj1 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for HTTP/127.27.209.212@KRBTEST.COM
I20260809 06:48:54.803426 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:48:54.845480 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.825822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.27.209.254:37983)
0809 06:48:54.826011 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.826017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.829549 (+ 3532us) server_negotiation.cc:407] Connection header received
0809 06:48:54.829811 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.829817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.829933 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.830049 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:54.831883 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.833028 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.834595 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.835256 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.835358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.838999 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:54.839006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:54.839011 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:54.839052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:54.842464 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.843313 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.843319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.843322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.843472 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:54.843930 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:54.843935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:54.843938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:54.844291 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:54.844533 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.844915 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.845148 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:48:54.851527 15694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:48:54.862893 15694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:48:54.961403 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.902613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53668)
0809 06:48:54.932305 (+ 29692us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:54.932326 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:54.932444 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:54.935015 (+ 2571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:54.935024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:54.935056 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:54.935466 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.935482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.939707 (+ 4225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.939713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.951686 (+ 11973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.951703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.959199 (+ 7496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.959206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.959429 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.959488 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 06:48:54.960813 (+ 1325us) client_negotiation.cc:240] Negotiation successful
0809 06:48:54.961035 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29556,"thread_start_us":29406,"threads_started":1}
I20260809 06:48:54.962855 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.894456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:45171 (local address 127.0.0.1:33288)
0809 06:48:54.921052 (+ 26596us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:54.921070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:54.921173 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:54.942719 (+ 21546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:54.942725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:54.942754 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:54.943075 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.943087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.953963 (+ 10876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.953969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.956464 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.956480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.962068 (+ 5588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.962074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.962245 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.962279 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:48:54.962357 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:48:54.962489 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26476,"thread_start_us":26906,"threads_started":1}
I20260809 06:48:54.964066 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.894854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.27.209.195:45171)
0809 06:48:54.936048 (+ 41194us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.936056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.936089 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:48:54.936179 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.936186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.936444 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.943313 (+ 6869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:54.943629 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.945294 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.957479 (+ 12185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.958977 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.963260 (+ 4283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.963449 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.963518 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.963728 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41020,"thread_start_us":40872,"threads_started":1}
I20260809 06:48:54.966176 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.903289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.27.209.193:45239)
0809 06:48:54.934011 (+ 30722us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.934021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.934053 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:48:54.934155 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.934164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.934474 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.934690 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:54.936727 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.939505 (+ 2778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.951910 (+ 12405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.953479 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.953667 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.953852 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.964445 (+ 10593us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.964645 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30541,"thread_start_us":29469,"threads_started":1}
I20260809 06:48:54.966888 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.891824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:37727 (local address 127.0.0.1:42704)
0809 06:48:54.920123 (+ 28299us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:54.920181 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:54.920458 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:54.948551 (+ 28093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:54.948557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:54.948585 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:54.948933 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.948945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.961741 (+ 12796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.961747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.964106 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:54.964122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.966129 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.966135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:54.966319 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.966350 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:48:54.966418 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 06:48:54.966533 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27983,"mutex_wait_us":1,"thread_start_us":29428,"threads_started":1}
I20260809 06:48:54.967882 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:54.892435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.27.209.194:37727)
0809 06:48:54.945783 (+ 53348us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:54.945792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:54.945823 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:48:54.945930 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:54.945939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:54.946179 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:54.949135 (+ 2956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:54.949394 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.950991 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.964411 (+ 13420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:54.965889 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:54.967155 (+ 1266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:54.967332 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:54.967398 (+ 66us) server_negotiation.cc:299] Negotiation successful
0809 06:48:54.967593 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53175,"thread_start_us":55122,"threads_started":1}
I20260809 06:48:54.968722 16307 tablet_service.cc:1467] Processing CreateTablet for tablet 00944cf951524b8d87f0d0a9c6fb468c (DEFAULT_TABLE table=test-table [id=872a6a3a5b0e42ae9e3c13c4c9af4c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:48:54.970424 16307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00944cf951524b8d87f0d0a9c6fb468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:54.974206 15883 tablet_service.cc:1467] Processing CreateTablet for tablet 00944cf951524b8d87f0d0a9c6fb468c (DEFAULT_TABLE table=test-table [id=872a6a3a5b0e42ae9e3c13c4c9af4c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:48:54.976825 16098 tablet_service.cc:1467] Processing CreateTablet for tablet 00944cf951524b8d87f0d0a9c6fb468c (DEFAULT_TABLE table=test-table [id=872a6a3a5b0e42ae9e3c13c4c9af4c90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:48:54.978147 16098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00944cf951524b8d87f0d0a9c6fb468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:54.979954 15883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00944cf951524b8d87f0d0a9c6fb468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:54.993070 16432 tablet_bootstrap.cc:492] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Bootstrap starting.
I20260809 06:48:54.998677 16432 tablet_bootstrap.cc:654] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:55.000082 16432 log.cc:824] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:55.009836 16432 tablet_bootstrap.cc:492] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: No bootstrap required, opened a new log
I20260809 06:48:55.010274 16432 ts_tablet_manager.cc:1397] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260809 06:48:55.021387 16432 raft_consensus.cc:348] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.022063 16432 raft_consensus.cc:374] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:55.022231 16432 raft_consensus.cc:729] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8f683525014e59aca8f83b007727d4, State: Initialized, Role: FOLLOWER
I20260809 06:48:55.023433 16432 consensus_queue.cc:260] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.024784 16436 tablet_bootstrap.cc:492] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Bootstrap starting.
I20260809 06:48:55.028905 16437 tablet_bootstrap.cc:492] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Bootstrap starting.
I20260809 06:48:55.029004 16436 tablet_bootstrap.cc:654] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:55.035230 16432 ts_tablet_manager.cc:1428] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Time spent starting tablet: real 0.025s user 0.016s sys 0.000s
I20260809 06:48:55.035729 16436 log.cc:824] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:55.036129 16403 heartbeater.cc:503] Master 127.27.209.254:37983 was elected leader, sending a full tablet report...
I20260809 06:48:55.037168 16437 tablet_bootstrap.cc:654] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:55.038468 16437 log.cc:824] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:55.045874 16436 tablet_bootstrap.cc:492] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: No bootstrap required, opened a new log
I20260809 06:48:55.046283 16436 ts_tablet_manager.cc:1397] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20260809 06:48:55.054131 16437 tablet_bootstrap.cc:492] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: No bootstrap required, opened a new log
I20260809 06:48:55.054446 16437 ts_tablet_manager.cc:1397] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20260809 06:48:55.058215 16436 raft_consensus.cc:348] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.058763 16436 raft_consensus.cc:374] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:55.058866 16436 raft_consensus.cc:729] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62974c8a94e141bb96e9b74737fd791e, State: Initialized, Role: FOLLOWER
I20260809 06:48:55.059497 16436 consensus_queue.cc:260] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.063900 16197 heartbeater.cc:503] Master 127.27.209.254:37983 was elected leader, sending a full tablet report...
W20260809 06:48:55.065235 16200 tablet.cc:2369] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:48:55.064972 16436 ts_tablet_manager.cc:1428] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260809 06:48:55.074788 16437 raft_consensus.cc:348] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.075157 16437 raft_consensus.cc:374] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:55.075260 16437 raft_consensus.cc:729] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f87aa0f0394efdb90ca868a0a891fa, State: Initialized, Role: FOLLOWER
I20260809 06:48:55.075925 16437 consensus_queue.cc:260] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.076993 15980 heartbeater.cc:503] Master 127.27.209.254:37983 was elected leader, sending a full tablet report...
I20260809 06:48:55.077548 16437 ts_tablet_manager.cc:1428] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Time spent starting tablet: real 0.023s user 0.011s sys 0.000s
W20260809 06:48:55.142458 15982 tablet.cc:2369] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:48:55.180514 16456 raft_consensus.cc:482] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:48:55.180851 16456 raft_consensus.cc:504] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.182796 16456 leader_election.cc:288] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62974c8a94e141bb96e9b74737fd791e (127.27.209.194:37727), 8b8f683525014e59aca8f83b007727d4 (127.27.209.195:45171)
I20260809 06:48:55.190888 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.183322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:35869 (local address 127.27.209.194:37727)
0809 06:48:55.183595 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:55.183603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:55.183892 (+ 289us) server_negotiation.cc:407] Connection header received
0809 06:48:55.184146 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:55.184152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:55.184258 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:55.184338 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:55.184973 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.185986 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.188609 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.189841 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.189970 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.190045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:55.190499 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 06:48:55.190623 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 06:48:55.190888 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.183413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:42415 (local address 127.27.209.195:45171)
0809 06:48:55.183767 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:55.183773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:55.184406 (+ 633us) server_negotiation.cc:407] Connection header received
0809 06:48:55.184569 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:55.184573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:55.184632 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:55.184726 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:48:55.185250 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.186241 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.188609 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.189812 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.189970 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.190045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:55.190497 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 06:48:55.190624 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260809 06:48:55.191154 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.183048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:37727 (local address 127.27.209.193:35869)
0809 06:48:55.183782 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.183802 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:55.183961 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:55.184380 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:55.184386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:55.184442 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:55.184804 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:55.184818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.186112 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.186116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:55.188453 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:55.188464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.190064 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.190075 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:55.190318 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.190366 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 06:48:55.190457 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 06:48:55.190649 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":138,"spinlock_wait_cycles":32768,"thread_start_us":981,"threads_started":1}
I20260809 06:48:55.191176 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.183083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:45171 (local address 127.27.209.193:42415)
0809 06:48:55.184295 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.184311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:55.184452 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:55.184747 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:55.184752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:55.184778 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:48:55.185118 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:55.185126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.186351 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.186356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:55.188448 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:55.188464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.190088 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.190093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:55.190313 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.190373 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 06:48:55.190458 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 06:48:55.190649 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"mutex_wait_us":92,"spinlock_wait_cycles":10112,"thread_start_us":344,"threads_started":1}
I20260809 06:48:55.192258 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00944cf951524b8d87f0d0a9c6fb468c" candidate_uuid: "11f87aa0f0394efdb90ca868a0a891fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8f683525014e59aca8f83b007727d4" is_pre_election: true
I20260809 06:48:55.192260 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00944cf951524b8d87f0d0a9c6fb468c" candidate_uuid: "11f87aa0f0394efdb90ca868a0a891fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62974c8a94e141bb96e9b74737fd791e" is_pre_election: true
I20260809 06:48:55.192742 16335 raft_consensus.cc:2454] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f87aa0f0394efdb90ca868a0a891fa in term 0.
I20260809 06:48:55.192749 16127 raft_consensus.cc:2454] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f87aa0f0394efdb90ca868a0a891fa in term 0.
I20260809 06:48:55.193362 15821 leader_election.cc:302] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f87aa0f0394efdb90ca868a0a891fa, 62974c8a94e141bb96e9b74737fd791e; no voters:
I20260809 06:48:55.193805 16456 raft_consensus.cc:2781] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:48:55.193960 16456 raft_consensus.cc:482] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:48:55.194061 16456 raft_consensus.cc:3037] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:55.196277 16456 raft_consensus.cc:504] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.196852 16456 leader_election.cc:288] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [CANDIDATE]: Term 1 election: Requested vote from peers 62974c8a94e141bb96e9b74737fd791e (127.27.209.194:37727), 8b8f683525014e59aca8f83b007727d4 (127.27.209.195:45171)
I20260809 06:48:55.197306 16127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00944cf951524b8d87f0d0a9c6fb468c" candidate_uuid: "11f87aa0f0394efdb90ca868a0a891fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62974c8a94e141bb96e9b74737fd791e"
I20260809 06:48:55.197376 16335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00944cf951524b8d87f0d0a9c6fb468c" candidate_uuid: "11f87aa0f0394efdb90ca868a0a891fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b8f683525014e59aca8f83b007727d4"
I20260809 06:48:55.197526 16127 raft_consensus.cc:3037] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:55.197570 16335 raft_consensus.cc:3037] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:55.200671 16127 raft_consensus.cc:2454] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f87aa0f0394efdb90ca868a0a891fa in term 1.
I20260809 06:48:55.200809 16335 raft_consensus.cc:2454] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11f87aa0f0394efdb90ca868a0a891fa in term 1.
I20260809 06:48:55.201144 15821 leader_election.cc:302] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f87aa0f0394efdb90ca868a0a891fa, 62974c8a94e141bb96e9b74737fd791e; no voters:
I20260809 06:48:55.201416 16456 raft_consensus.cc:2781] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:48:55.201838 16456 raft_consensus.cc:686] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 1 LEADER]: Becoming Leader. State: Replica: 11f87aa0f0394efdb90ca868a0a891fa, State: Running, Role: LEADER
I20260809 06:48:55.202301 16456 consensus_queue.cc:237] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } }
I20260809 06:48:55.206983 15695 catalog_manager.cc:5495] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 11f87aa0f0394efdb90ca868a0a891fa (127.27.209.193). New cstate: current_term: 1 leader_uuid: "11f87aa0f0394efdb90ca868a0a891fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f87aa0f0394efdb90ca868a0a891fa" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 45239 } health_report { overall_health: HEALTHY } } }
I20260809 06:48:55.231148 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.226186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.27.209.193:45239)
0809 06:48:55.226373 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:55.226377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:55.226394 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:48:55.226520 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:55.226524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:55.226597 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:55.226737 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:48:55.227003 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.227768 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.229172 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.229635 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.229760 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.230180 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:48:55.230655 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:48:55.230801 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:55.230838 (+ 37us) server_negotiation.cc:299] Negotiation successful
0809 06:48:55.230899 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:48:55.239265 16127 raft_consensus.cc:1261] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 1 FOLLOWER]: Refusing update from remote peer 11f87aa0f0394efdb90ca868a0a891fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:48:55.239265 16335 raft_consensus.cc:1261] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 1 FOLLOWER]: Refusing update from remote peer 11f87aa0f0394efdb90ca868a0a891fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:48:55.240301 16459 consensus_queue.cc:1035] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "62974c8a94e141bb96e9b74737fd791e" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:48:55.240666 16456 consensus_queue.cc:1035] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b8f683525014e59aca8f83b007727d4" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:48:55.253235 16468 mvcc.cc:205] Tried to move back new op lower bound from 7316513321929977856 to 7316513321792331776. Current Snapshot: MvccSnapshot[applied={T|T < 7316513321929977856}]
I20260809 06:48:55.265820 15694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:48:55.266093 15694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:48:55.269603 15694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 893bfbfdaa0c4335850d0025ceb68e94: Sending DeleteTablet for 3 replicas of tablet 00944cf951524b8d87f0d0a9c6fb468c
I20260809 06:48:55.270650 16098 tablet_service.cc:1514] Processing DeleteTablet for tablet 00944cf951524b8d87f0d0a9c6fb468c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:48:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42704
I20260809 06:48:55.270738 16307 tablet_service.cc:1514] Processing DeleteTablet for tablet 00944cf951524b8d87f0d0a9c6fb468c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:48:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:33288
I20260809 06:48:55.271597 16477 tablet_replica.cc:330] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: stopping tablet replica
I20260809 06:48:55.271576 15883 tablet_service.cc:1514] Processing DeleteTablet for tablet 00944cf951524b8d87f0d0a9c6fb468c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:48:55 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:53668
I20260809 06:48:55.272032 16477 raft_consensus.cc:2229] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:48:55.272521 16477 raft_consensus.cc:2258] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:48:55.276559 16477 ts_tablet_manager.cc:1905] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:48:55.277897 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 15748
I20260809 06:48:55.284313 16477 ts_tablet_manager.cc:1918] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:48:55.284596 16477 log.cc:1196] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/00944cf951524b8d87f0d0a9c6fb468c
I20260809 06:48:55.285207 16477 ts_tablet_manager.cc:1939] T 00944cf951524b8d87f0d0a9c6fb468c P 8b8f683525014e59aca8f83b007727d4: Deleting consensus metadata
I20260809 06:48:55.287220 15670 catalog_manager.cc:4841] TS 8b8f683525014e59aca8f83b007727d4 (127.27.209.195:45171): tablet 00944cf951524b8d87f0d0a9c6fb468c (table test-table [id=872a6a3a5b0e42ae9e3c13c4c9af4c90]) successfully deleted
I20260809 06:48:55.296130 16481 tablet_replica.cc:330] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: stopping tablet replica
I20260809 06:48:55.296617 16481 raft_consensus.cc:2229] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:48:55.297293 16481 raft_consensus.cc:2258] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:48:55.298136 16480 tablet_replica.cc:330] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: stopping tablet replica
I20260809 06:48:55.298556 16480 raft_consensus.cc:2229] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:48:55.299266 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.280606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.27.209.193:45239)
0809 06:48:55.280796 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:55.280802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:55.280823 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:48:55.280884 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:55.280889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:55.280988 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:55.281097 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:55.283200 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.284554 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.286651 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.287350 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.287478 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.291089 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:55.291104 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:55.291124 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:55.291173 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:55.295519 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:55.296638 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:55.296649 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:55.296653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:55.296952 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:55.297495 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:55.297500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:55.297504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:55.298001 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:55.298358 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:55.298657 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 06:48:55.298885 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:48:55.299657 16480 raft_consensus.cc:2258] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:48:55.300256 15973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:55.302242 16480 ts_tablet_manager.cc:1905] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:48:55.307909 16480 ts_tablet_manager.cc:1918] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:48:55.308045 16480 log.cc:1196] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/00944cf951524b8d87f0d0a9c6fb468c
I20260809 06:48:55.308461 16480 ts_tablet_manager.cc:1939] T 00944cf951524b8d87f0d0a9c6fb468c P 62974c8a94e141bb96e9b74737fd791e: Deleting consensus metadata
I20260809 06:48:55.309948 15675 catalog_manager.cc:4841] TS 62974c8a94e141bb96e9b74737fd791e (127.27.209.194:37727): tablet 00944cf951524b8d87f0d0a9c6fb468c (table test-table [id=872a6a3a5b0e42ae9e3c13c4c9af4c90]) successfully deleted
I20260809 06:48:55.300678 16481 ts_tablet_manager.cc:1905] T 00944cf951524b8d87f0d0a9c6fb468c P 11f87aa0f0394efdb90ca868a0a891fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:48:55.397514 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 15748
W20260809 06:48:55.419947 15674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.209.193:45239 (error 108)
I20260809 06:48:55.420648 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 15987
I20260809 06:48:55.421622 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.421147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53692)
0809 06:48:55.421346 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.421470 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 06:48:55.421838 15674 catalog_manager.cc:4584] TS 11f87aa0f0394efdb90ca868a0a891fa (127.27.209.193:45239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00944cf951524b8d87f0d0a9c6fb468c: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
I20260809 06:48:55.433532 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.421431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.27.209.194:37727)
0809 06:48:55.421625 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:55.421631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:55.421650 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:48:55.421719 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:55.421723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:55.421809 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:55.421914 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:55.423191 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.423973 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.425144 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.425539 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.425632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.428219 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:55.428256 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:55.428272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:55.428309 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:55.430909 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:55.431555 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:55.431565 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:55.431569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:55.431712 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:55.432013 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:55.432016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:55.432018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:55.432364 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:55.432644 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:55.432968 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 06:48:55.433200 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:48:55.434206 16189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:55.475587 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.474993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53708)
0809 06:48:55.475239 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.475376 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 06:48:55.476660 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.476263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53710)
0809 06:48:55.476412 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.476518 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 06:48:55.521847 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 15987
I20260809 06:48:55.537194 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.536724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53722)
0809 06:48:55.536923 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.537019 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:48:55.538192 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.537850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53728)
0809 06:48:55.538007 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.538078 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 06:48:55.549934 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 16208
I20260809 06:48:55.567493 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.550796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.27.209.195:45171)
0809 06:48:55.551015 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:55.551021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:55.551043 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:48:55.551255 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:55.551261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:55.551335 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:55.551438 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:55.552881 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.553681 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.555037 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:55.555479 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:55.555630 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:55.562145 (+ 6515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:55.562160 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:55.562179 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:55.562214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:55.564811 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:55.565467 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:55.565475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:55.565477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:55.565641 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:55.565988 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:55.565992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:55.565994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:55.566364 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:55.566665 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:55.566947 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 06:48:55.567157 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 06:48:55.568100 16395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:55.619161 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.618520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53742)
0809 06:48:55.618803 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.618932 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 06:48:55.620272 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:55.619839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:45239 (local address 127.0.0.1:53750)
0809 06:48:55.620043 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:55.620132 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:45239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 06:48:55.654928 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 16208
I20260809 06:48:55.678115 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 15642
I20260809 06:48:55.679334 15740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:48:55.793130 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 15642
I20260809 06:48:55.821583 28487 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 06:48:56.155830 28487 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 06:48:56.156059 28487 mini_ranger.cc:66] Stopping Ranger...
I20260809 06:48:56.500260 28487 mini_ranger.cc:68] Stopped Ranger
2026-08-09 06:48:56.500 UTC [14196] LOG: received smart shutdown request
2026-08-09 06:48:56.503 UTC [14196] LOG: background worker "logical replication launcher" (PID 14211) exited with exit code 1
2026-08-09 06:48:56.503 UTC [14203] LOG: shutting down
2026-08-09 06:48:56.643 UTC [14196] LOG: database system is shut down
2026-08-09T06:48:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51394 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:48:56 dist-test-slave-jpj1 krb5kdc[16499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:48:56 dist-test-slave-jpj1 krb5kdc[16499](info): set up 2 sockets
Aug 09 06:48:56 dist-test-slave-jpj1 krb5kdc[16499](info): commencing operation
krb5kdc: starting...
W20260809 06:48:57.956012 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:48:58 dist-test-slave-jpj1 krb5kdc[16499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:48:58.206969 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38147
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:38147,127.27.209.253:43687,127.27.209.252:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:48:58.533502 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:58.533810 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:58.533946 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:58.541150 16520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:48:58.541257 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:58.541332 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:58.541410 16520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:48:58.541476 16520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:48:58.551620 16520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:38147,127.27.209.253:43687,127.27.209.252:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:58.552983 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:58.554340 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:58.565630 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:58.565678 16530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:58.565709 16529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:58.566419 16520 server_base.cc:1034] running on GCE node
I20260809 06:48:58.567232 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:58.568619 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:58.569854 16520 hybrid_clock.cc:648] HybridClock initialized: now 1786258138569788 us; error 46 us; skew 500 ppm
I20260809 06:48:58.570200 16520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:58 dist-test-slave-jpj1 krb5kdc[16499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:58.573346 16520 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:48:58.574802 16520 webserver.cc:460] Webserver started at http://127.27.209.254:33799/ using document root <none> and password file <none>
I20260809 06:48:58.575465 16520 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:58.575610 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:58.575927 16520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:48:58.578891 16520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "43d1356cd97c420bbeb10d6e113cb1b6"
format_stamp: "Formatted at 2026-08-09 06:48:58 on dist-test-slave-jpj1"
server_key: "4da66933575b0c47c8df719e0d2e44ac"
server_key_iv: "cd7abbd94cbc4eda3460e805d5f8a3ed"
server_key_version: "encryptionkey@0"
I20260809 06:48:58.579764 16520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "43d1356cd97c420bbeb10d6e113cb1b6"
format_stamp: "Formatted at 2026-08-09 06:48:58 on dist-test-slave-jpj1"
server_key: "4da66933575b0c47c8df719e0d2e44ac"
server_key_iv: "cd7abbd94cbc4eda3460e805d5f8a3ed"
server_key_version: "encryptionkey@0"
I20260809 06:48:58.584824 16520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:48:58.588150 16545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:58.589429 16520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:48:58.589599 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "43d1356cd97c420bbeb10d6e113cb1b6"
format_stamp: "Formatted at 2026-08-09 06:48:58 on dist-test-slave-jpj1"
server_key: "4da66933575b0c47c8df719e0d2e44ac"
server_key_iv: "cd7abbd94cbc4eda3460e805d5f8a3ed"
server_key_version: "encryptionkey@0"
I20260809 06:48:58.589807 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:58.621143 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:58.625720 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:58.626019 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:58.644238 16520 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:38147
I20260809 06:48:58.644289 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:38147 every 8 connection(s)
I20260809 06:48:58.645573 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:48:58.649180 16617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:58.649902 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 16520
I20260809 06:48:58.650166 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:48:58.651041 28487 external_mini_cluster.cc:1428] Setting key 678c43197d71266de2f55bb427046e86
I20260809 06:48:58.657975 16617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:48:58 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.209.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.253@KRBTEST.COM" created.
I20260809 06:48:58.679323 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.661519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38147 (local address 127.0.0.1:51858)
0809 06:48:58.662273 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.662312 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:58.664318 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:58.665112 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:58.665119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:58.665818 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:58.666145 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:58.666163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:58.667692 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:58.667696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:58.668745 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:58.668767 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:58.669660 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:58.669668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:58.669849 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:58.670478 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:58.670499 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:58.672666 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:58.675693 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:58.675703 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:58.675716 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:58.676090 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:58.676516 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:58.676520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:58.676523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:58.676704 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:58.677490 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:58.677497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:58.677687 (+ 190us) client_negotiation.cc:769] Sending connection context
0809 06:48:58.677902 (+ 215us) client_negotiation.cc:240] Negotiation successful
0809 06:48:58.678297 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":320,"threads_started":1}
I20260809 06:48:58.679382 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.662824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.27.209.254:38147)
0809 06:48:58.663558 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:58.663568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:58.663618 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:48:58.664659 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:58.664678 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:58.664896 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:58.665068 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:58.666343 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:58.667526 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:58.668885 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:58.669424 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:58.669615 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:58.672872 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:58.672881 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:58.672892 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:58.672920 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:58.675588 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:58.676207 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:58.676217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:58.676221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:58.676374 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:58.676804 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:58.676809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:58.676812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:58.677347 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:58.677545 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:58.678070 (+ 525us) server_negotiation.cc:299] Negotiation successful
0809 06:48:58.678482 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":486,"thread_start_us":303,"threads_started":1}
I20260809 06:48:58.682399 16617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:48:58.683435 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.683066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33386)
0809 06:48:58.683228 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.683337 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 06:48:58.683768 16549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
I20260809 06:48:58.685331 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.684958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33402)
0809 06:48:58.685102 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.685200 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 06:48:58.685516 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:43687: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab.
Entry for principal kudu/127.27.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab.
I20260809 06:48:58.705021 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:43687
--webserver_interface=127.27.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab
--principal=kudu/127.27.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:38147,127.27.209.253:43687,127.27.209.252:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:48:58.734822 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } attempt: 1
I20260809 06:48:58.736346 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.735646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33414)
0809 06:48:58.735961 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.736148 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260809 06:48:58.737490 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.737129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33424)
0809 06:48:58.737286 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.737362 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 06:48:58.737659 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:43687: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
I20260809 06:48:58.805902 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } attempt: 2
I20260809 06:48:58.806996 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.806522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33434)
0809 06:48:58.806764 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.806856 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:48:58.808039 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.807681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33438)
0809 06:48:58.807831 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.807912 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 06:48:58.808204 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:43687: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
I20260809 06:48:58.899435 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } attempt: 3
I20260809 06:48:58.900557 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.900129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33444)
0809 06:48:58.900333 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.900424 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 06:48:58.901569 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:58.901211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33454)
0809 06:48:58.901357 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:58.901436 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 06:48:58.901775 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:43687: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
W20260809 06:48:59.028832 16628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:59.029137 16628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:59.029277 16628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:59.036466 16628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:48:59.036583 16628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:59.036664 16628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:59.036742 16628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:48:59.036809 16628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:48:59.045001 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } attempt: 4
I20260809 06:48:59.046142 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.045596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33464)
0809 06:48:59.045912 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.046001 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260809 06:48:59.046829 16628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:38147,127.27.209.253:43687,127.27.209.252:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab
--principal=kudu/127.27.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:59.047178 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.046805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33468)
0809 06:48:59.046946 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.047042 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:48:59.047343 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:43687: Network error: Client connection negotiation failed: client connection to 127.27.209.253:43687: connect: Connection refused (error 111)
I20260809 06:48:59.048237 16628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:59.049532 16628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:48:59.060961 16638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:59.060971 16636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:59.061013 16635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:59.062000 16628 server_base.cc:1034] running on GCE node
I20260809 06:48:59.062762 16628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:59.064142 16628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:59.065353 16628 hybrid_clock.cc:648] HybridClock initialized: now 1786258139065311 us; error 30 us; skew 500 ppm
I20260809 06:48:59.065690 16628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:59.068711 16628 init.cc:377] Logged in from keytab as kudu/127.27.209.253@KRBTEST.COM (short username kudu)
I20260809 06:48:59.070166 16628 webserver.cc:460] Webserver started at http://127.27.209.253:44167/ using document root <none> and password file <none>
I20260809 06:48:59.070845 16628 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:59.070961 16628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:59.071277 16628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:48:59.074505 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data/instance:
uuid: "617f92beb5b54a5cb28341e328e49978"
format_stamp: "Formatted at 2026-08-09 06:48:59 on dist-test-slave-jpj1"
server_key: "d5ca19eab3fe120316a8fc3f68df6708"
server_key_iv: "db005e9bafaa17205ffc9c5e241c07cc"
server_key_version: "encryptionkey@0"
I20260809 06:48:59.075344 16628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal/instance:
uuid: "617f92beb5b54a5cb28341e328e49978"
format_stamp: "Formatted at 2026-08-09 06:48:59 on dist-test-slave-jpj1"
server_key: "d5ca19eab3fe120316a8fc3f68df6708"
server_key_iv: "db005e9bafaa17205ffc9c5e241c07cc"
server_key_version: "encryptionkey@0"
I20260809 06:48:59.080714 16628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 06:48:59.084257 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:59.085541 16628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:48:59.085704 16628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal
uuid: "617f92beb5b54a5cb28341e328e49978"
format_stamp: "Formatted at 2026-08-09 06:48:59 on dist-test-slave-jpj1"
server_key: "d5ca19eab3fe120316a8fc3f68df6708"
server_key_iv: "db005e9bafaa17205ffc9c5e241c07cc"
server_key_version: "encryptionkey@0"
I20260809 06:48:59.085883 16628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:59.122819 16628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:59.126677 16628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:59.126947 16628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:59.144202 16628 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.253:43687
I20260809 06:48:59.144225 16721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.253:43687 every 8 connection(s)
I20260809 06:48:59.145534 16628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
I20260809 06:48:59.147575 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 16628
I20260809 06:48:59.147774 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-1/wal/instance
I20260809 06:48:59.148370 28487 external_mini_cluster.cc:1428] Setting key ffe033c099d438293c82d61542f54d22
I20260809 06:48:59.149940 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:59.158444 16722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.252@KRBTEST.COM" created.
I20260809 06:48:59.178875 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.161291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.27.209.254:38147)
0809 06:48:59.161458 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.161462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.162745 (+ 1283us) server_negotiation.cc:407] Connection header received
0809 06:48:59.164601 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.164608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.164688 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.164775 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.166248 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.166925 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.168464 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.168959 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.169088 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.173020 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.173026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.173029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.173063 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.175725 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.176627 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.176632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.176636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.176791 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.177170 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.177176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.177180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.177559 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.177714 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.178352 (+ 638us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.178576 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:48:59.179772 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.161852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38147 (local address 127.0.0.1:51860)
0809 06:48:59.162590 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.162625 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.164337 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.165009 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.165017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.165511 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.166057 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.166074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.167074 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.167077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.168324 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.168337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.169222 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.169230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.169525 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.170244 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.170281 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.172799 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.175868 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.175875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.175888 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.176490 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.176896 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.176899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.176901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.177063 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.177667 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.177673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.177994 (+ 321us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.178225 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.178611 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":302,"threads_started":1}
I20260809 06:48:59.181502 16722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:48:59.195999 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.182266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33474)
0809 06:48:59.182448 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.182465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.182564 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.183553 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.183562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.183979 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.184141 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.184152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.185469 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.185474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.186327 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.186338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.187374 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.187379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.187561 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.188087 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.188102 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.190081 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.193714 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.193718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.193720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.194138 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.194504 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.194507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.194509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.194644 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.195301 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.195304 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.195411 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.195514 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.195736 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":3}
I20260809 06:48:59.196321 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.182484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.27.209.253:43687)
0809 06:48:59.183116 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.183122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.183148 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:48:59.183191 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.183196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.183378 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.183486 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.184301 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.185330 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.186483 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.187236 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.187361 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.190305 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.190318 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.190334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.190367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.193576 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.194259 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.194268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.194270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.194402 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.194737 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.194741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.194744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.195174 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.195406 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.195694 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.195928 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":346,"threads_started":1}
I20260809 06:48:59.198879 16722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:48:59.199939 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.199504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55570)
0809 06:48:59.199692 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.199802 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 06:48:59.200217 16652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.202103 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.201782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55576)
0809 06:48:59.201930 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.202004 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 06:48:59.202306 16722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab.
Entry for principal kudu/127.27.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab.
I20260809 06:48:59.208580 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:44467
--webserver_interface=127.27.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab
--principal=kudu/127.27.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:38147,127.27.209.253:43687,127.27.209.252:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:48:59.251644 16722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 1
I20260809 06:48:59.252935 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.252451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55580)
0809 06:48:59.252656 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.252779 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 06:48:59.253906 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.253562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55590)
0809 06:48:59.253708 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.253787 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:48:59.254150 16722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.322360 16722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 2
I20260809 06:48:59.323513 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.323070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55604)
0809 06:48:59.323262 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.323355 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:48:59.324484 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.324131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55608)
0809 06:48:59.324279 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.324354 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 06:48:59.324676 16722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.346577 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } attempt: 5
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:48:59.360415 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.347292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33480)
0809 06:48:59.347474 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.347495 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.347640 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.348072 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.348076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.348440 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.348620 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.348629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.349982 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.349986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.350831 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.350851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.351849 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.351853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.351988 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.352581 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.352599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.354605 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.358209 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.358213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.358216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.358563 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.358957 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.358960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.358962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.359089 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.359692 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.359695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.359818 (+ 123us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.359935 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.360154 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 06:48:59.360576 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.347390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.27.209.253:43687)
0809 06:48:59.347603 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.347609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.347634 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:48:59.347801 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.347806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.347909 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.348029 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.348809 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.349869 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.350993 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.351697 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.351847 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.354823 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.354830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.354835 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.354875 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.358059 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.358681 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.358687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.358691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.358859 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.359189 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.359195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.359198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.359596 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.359742 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.360054 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.360284 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:48:59.361594 16617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:48:59.362529 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.362179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55622)
0809 06:48:59.362333 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.362406 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 06:48:59.363432 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.363125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55632)
0809 06:48:59.363268 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.363332 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:48:59.363639 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.414844 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 1
I20260809 06:48:59.415987 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.415466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55634)
0809 06:48:59.415681 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.415797 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 06:48:59.415897 16722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 3
I20260809 06:48:59.416854 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.416466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55640)
0809 06:48:59.416627 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.416707 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 06:48:59.417191 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.416795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55642)
0809 06:48:59.416969 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.417067 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 06:48:59.417490 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.418002 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.417575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55644)
0809 06:48:59.417800 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.417893 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260809 06:48:59.418187 16722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.485715 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 2
I20260809 06:48:59.486912 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.486430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55650)
0809 06:48:59.486644 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.486732 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 06:48:59.487921 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.487604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55656)
0809 06:48:59.487752 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.487819 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 06:48:59.488113 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
W20260809 06:48:59.541643 16739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:48:59.541939 16739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:48:59.542080 16739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:48:59.549024 16739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:48:59.549150 16739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:48:59.549233 16739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:48:59.549325 16739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:48:59.549396 16739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:48:59.558897 16739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:38147,127.27.209.253:43687,127.27.209.252:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab
--principal=kudu/127.27.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:44467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:48:59.560230 16739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:48:59.561390 16722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 4
I20260809 06:48:59.561589 16739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 06:48:59.562546 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.562067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55664)
0809 06:48:59.562301 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.562397 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:48:59.563524 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.563177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55666)
0809 06:48:59.563315 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.563382 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:48:59.563800 16722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
W20260809 06:48:59.573364 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:59.573328 16749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:48:59.573278 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:48:59.574191 16739 server_base.cc:1034] running on GCE node
I20260809 06:48:59.575006 16739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:48:59.576467 16739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:48:59.577685 16739 hybrid_clock.cc:648] HybridClock initialized: now 1786258139577615 us; error 57 us; skew 500 ppm
I20260809 06:48:59.578037 16739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:48:59.581089 16739 init.cc:377] Logged in from keytab as kudu/127.27.209.252@KRBTEST.COM (short username kudu)
I20260809 06:48:59.582612 16739 webserver.cc:460] Webserver started at http://127.27.209.252:33483/ using document root <none> and password file <none>
I20260809 06:48:59.583305 16739 fs_manager.cc:359] Metadata directory not provided
I20260809 06:48:59.583422 16739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:48:59.583828 16739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:48:59.586813 16739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data/instance:
uuid: "23bca1bea0034184b7b79472ff6b7ce5"
format_stamp: "Formatted at 2026-08-09 06:48:59 on dist-test-slave-jpj1"
server_key: "17d4ce4f4cd784e49eee8c38989c770d"
server_key_iv: "f36dba1f43f3ec45cb25b02666606f59"
server_key_version: "encryptionkey@0"
I20260809 06:48:59.587622 16739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal/instance:
uuid: "23bca1bea0034184b7b79472ff6b7ce5"
format_stamp: "Formatted at 2026-08-09 06:48:59 on dist-test-slave-jpj1"
server_key: "17d4ce4f4cd784e49eee8c38989c770d"
server_key_iv: "f36dba1f43f3ec45cb25b02666606f59"
server_key_version: "encryptionkey@0"
I20260809 06:48:59.592895 16739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:48:59.594295 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 3
I20260809 06:48:59.595268 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.594813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55680)
0809 06:48:59.595018 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.595099 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 06:48:59.596338 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.595926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55694)
0809 06:48:59.596099 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.596206 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:48:59.596531 16617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:44467: Network error: Client connection negotiation failed: client connection to 127.27.209.252:44467: connect: Connection refused (error 111)
I20260809 06:48:59.596568 16760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:48:59.597855 16739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:48:59.598021 16739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal
uuid: "23bca1bea0034184b7b79472ff6b7ce5"
format_stamp: "Formatted at 2026-08-09 06:48:59 on dist-test-slave-jpj1"
server_key: "17d4ce4f4cd784e49eee8c38989c770d"
server_key_iv: "f36dba1f43f3ec45cb25b02666606f59"
server_key_version: "encryptionkey@0"
I20260809 06:48:59.598194 16739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:48:59.614253 16739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:48:59.618005 16739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:48:59.618266 16739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:48:59.634819 16739 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.252:44467
I20260809 06:48:59.634842 16835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.252:44467 every 8 connection(s)
I20260809 06:48:59.636221 16739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
I20260809 06:48:59.639941 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:48:59.643664 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 16739
I20260809 06:48:59.643862 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786257935933888-28487-0/minicluster-data/master-2/wal/instance
I20260809 06:48:59.644436 28487 external_mini_cluster.cc:1428] Setting key 3dfee46566fdaeceb4c4a612b2b65d27
I20260809 06:48:59.647073 16836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:48:59.672539 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.651008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.27.209.254:38147)
0809 06:48:59.651206 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.651213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.656356 (+ 5143us) server_negotiation.cc:407] Connection header received
0809 06:48:59.658522 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.658528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.658601 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.658692 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.659990 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.660767 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.662436 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.663065 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.663193 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.666684 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.666691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.666696 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.666736 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.669338 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.670196 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.670200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.670202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.670343 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.670820 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.670824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.670828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.671069 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.671180 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.672051 (+ 871us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.672268 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:48:59.673492 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.650513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38147 (local address 127.0.0.1:51874)
0809 06:48:59.655771 (+ 5258us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.655826 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.658279 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.658790 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.658801 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.659450 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.659818 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.659836 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.660925 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.660930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.662303 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.662315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.663274 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.663281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.663518 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.664289 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.664317 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.666482 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.669461 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.669472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.669488 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.670084 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.670460 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.670466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.670469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.670694 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.671196 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.671203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.671656 (+ 453us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.671889 (+ 233us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.672244 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4929,"spinlock_wait_cycles":16768,"thread_start_us":4778,"threads_started":1}
I20260809 06:48:59.674381 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.646233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.27.209.252:44467)
0809 06:48:59.656647 (+ 10414us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.656655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.657046 (+ 391us) server_negotiation.cc:407] Connection header received
0809 06:48:59.658280 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.658304 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.658528 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.658696 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.660192 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.661671 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.663289 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.664044 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.664161 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.668448 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.668456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.668468 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.668496 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.671490 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.672243 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.672258 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.672261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.672420 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.672861 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.672864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.672867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.673181 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.673456 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.673852 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.674051 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10192,"thread_start_us":9970,"threads_started":1}
I20260809 06:48:59.675309 16836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:48:59.688297 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.676145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:43687 (local address 127.0.0.1:33488)
0809 06:48:59.676314 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.676332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.676448 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.676805 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.676810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.677196 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.677370 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.677382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.678473 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.678479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.679498 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.679510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.680359 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.680366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.680556 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.681074 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.681095 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.683217 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.686090 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.686094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.686126 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.686465 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.686845 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.686848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.686851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.686985 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.687565 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.687571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.687750 (+ 179us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.687871 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.688056 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:48:59.688572 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.676125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.27.209.253:43687)
0809 06:48:59.676326 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.676331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.676390 (+ 59us) server_negotiation.cc:407] Connection header received
0809 06:48:59.676567 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.676572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.676650 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.676746 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.677525 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.678332 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.679650 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.680198 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.680299 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.683400 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.683405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.683408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.683441 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.685976 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.686581 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.686587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.686589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.686728 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.687092 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.687097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.687100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.687427 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.687604 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.687987 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.688196 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:48:59.689525 16836 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:48:59.691942 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.677429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.27.209.254:38147)
0809 06:48:59.677583 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.677588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.677603 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:48:59.677647 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.677652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.677713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.677805 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.679002 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.679904 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.681562 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.681957 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.682048 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.686105 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.686111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.686116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.686153 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.689192 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.689961 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.689964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.689967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.690102 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.690609 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.690614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.690618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.690973 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.691267 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.691446 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.691709 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:48:59.702971 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.690276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55706)
0809 06:48:59.690469 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.690487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.690616 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.690938 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.690942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.691391 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.691589 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.691602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.692845 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.692850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.693831 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.693843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.695013 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.695017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.695160 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.695715 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.695753 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.697940 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.700696 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.700700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.700705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.701079 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.701403 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.701408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.701412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.701546 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.702049 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.702053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.702191 (+ 138us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.702322 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.702567 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 06:48:59.703228 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.690471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.27.209.252:44467)
0809 06:48:59.690605 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.690608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.690623 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:48:59.690720 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.690723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.690810 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.690877 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.691760 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.692694 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.694282 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.694859 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.694981 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.698117 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.698124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.698129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.698163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.700582 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.701178 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.701183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.701187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.701310 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.701654 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.701659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.701662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.701946 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.702071 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.702407 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.702780 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":39}
I20260809 06:48:59.708400 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.694014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.27.209.253:43687)
0809 06:48:59.694170 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.694175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.694192 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:48:59.694359 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.694362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.694418 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.694490 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.695838 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.696881 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.698117 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.698532 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.698640 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.702730 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.702735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.702738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.702766 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.706052 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.706648 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.706654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.706656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.706821 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.707110 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.707115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.707118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.707423 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.707719 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.707889 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.708098 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:48:59.709142 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: Bootstrap starting.
I20260809 06:48:59.712803 16836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:59.713228 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:48:59.713927 16836 log.cc:824] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:59.716781 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: No bootstrap required, opened a new log
I20260809 06:48:59.727677 16836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.728259 16836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:59.728453 16836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bca1bea0034184b7b79472ff6b7ce5, State: Initialized, Role: FOLLOWER
I20260809 06:48:59.729307 16836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.736438 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.737615 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.718015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.27.209.254:38147)
0809 06:48:59.718182 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.718188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.720083 (+ 1895us) server_negotiation.cc:407] Connection header received
0809 06:48:59.720236 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.720241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.720324 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.720418 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.722811 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.723881 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.725823 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.726451 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.726542 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.730528 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.730535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.730540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.730593 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.734085 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.735092 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.735098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.735101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.735284 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.735814 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.735819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.735823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.736162 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.736393 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.737017 (+ 624us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.737272 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 06:48:59.738047 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.717745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.27.209.253:43687)
0809 06:48:59.717919 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.717925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.720680 (+ 2755us) server_negotiation.cc:407] Connection header received
0809 06:48:59.720924 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.720929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.721001 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.721066 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.723580 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.724585 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.726524 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.727108 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.727202 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.730713 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.730719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.730722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.730757 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.734120 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.735212 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.735217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.735220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.735419 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.735814 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.735819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.735823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.736182 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.736389 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.737385 (+ 996us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.737732 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:48:59.739198 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:48:59.739068 16836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:48:59.747270 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.720788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.27.209.252:44467)
0809 06:48:59.720968 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.720973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.728044 (+ 7071us) server_negotiation.cc:407] Connection header received
0809 06:48:59.728300 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.728306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.728380 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.728457 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.730809 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.731950 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.733990 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.734697 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.734801 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.741985 (+ 7184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.741990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.741994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.742024 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.744612 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.745489 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.745492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.745495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.745623 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.746009 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.746011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.746013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.746310 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.746480 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.746834 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.746990 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 06:48:59.756361 16871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:48:59.756490 16871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:48:59.752511 16739 master_runner.cc:186] Error getting master registration for 127.27.209.254:38147: OK, instance_id { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" instance_seqno: 1786258138639086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:48:59.763366 16739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:48:59.773710 16617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 4
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258138, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:48:59.788045 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.774259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.27.209.252:44467)
0809 06:48:59.774417 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.774421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.774456 (+ 35us) server_negotiation.cc:407] Connection header received
0809 06:48:59.774619 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.774622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.774689 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.774762 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.775719 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.776491 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.778186 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.778625 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.778718 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.782445 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.782452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.782456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.782493 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.785454 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.786227 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.786231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.786237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.786378 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.786766 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.786769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.786771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.787055 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.787158 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.787636 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.787800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:48:59.788275 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.774181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55726)
0809 06:48:59.774383 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.774405 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.774521 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.774815 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.774819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.775349 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.775590 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.775604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.776619 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.776625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.778057 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.778074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.778781 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.778788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.778982 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.779604 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.779629 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.782220 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.785583 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.785588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.785593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.786096 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.786490 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.786495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.786499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.786667 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.787158 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.787163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.787347 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.787513 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.787809 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:48:59.793279 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: Bootstrap starting.
I20260809 06:48:59.796520 16617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:59.797510 16617 log.cc:824] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:59.800093 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: No bootstrap required, opened a new log
I20260809 06:48:59.807624 16617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.807938 16617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:59.808041 16617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d1356cd97c420bbeb10d6e113cb1b6, State: Initialized, Role: FOLLOWER
I20260809 06:48:59.808544 16617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.809449 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.809870 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:48:59.810592 16617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:48:59.824497 16855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:48:59.824797 16855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
W20260809 06:48:59.825871 16894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:48:59.825986 16894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:48:59.826277 16855 leader_election.cc:288] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43d1356cd97c420bbeb10d6e113cb1b6 (127.27.209.254:38147), 617f92beb5b54a5cb28341e328e49978 (127.27.209.253:43687)
I20260809 06:48:59.827983 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23bca1bea0034184b7b79472ff6b7ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" is_pre_election: true
I20260809 06:48:59.828579 16583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bca1bea0034184b7b79472ff6b7ce5 in term 0.
I20260809 06:48:59.829209 16762 leader_election.cc:302] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23bca1bea0034184b7b79472ff6b7ce5, 43d1356cd97c420bbeb10d6e113cb1b6; no voters:
I20260809 06:48:59.829705 16855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:48:59.829851 16855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:48:59.829954 16855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:59.820135 16520 master_runner.cc:186] Error getting master registration for 127.27.209.253:43687: OK, instance_id { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" instance_seqno: 1786258139139209 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:48:59.831125 16520 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:48:59.834364 16855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.835456 16583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23bca1bea0034184b7b79472ff6b7ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43d1356cd97c420bbeb10d6e113cb1b6"
I20260809 06:48:59.835680 16583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:59.836349 16855 leader_election.cc:288] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [CANDIDATE]: Term 1 election: Requested vote from peers 43d1356cd97c420bbeb10d6e113cb1b6 (127.27.209.254:38147), 617f92beb5b54a5cb28341e328e49978 (127.27.209.253:43687)
I20260809 06:48:59.838845 16583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bca1bea0034184b7b79472ff6b7ce5 in term 1.
I20260809 06:48:59.839375 16762 leader_election.cc:302] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23bca1bea0034184b7b79472ff6b7ce5, 43d1356cd97c420bbeb10d6e113cb1b6; no voters:
I20260809 06:48:59.839697 16855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:48:59.847764 16855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 23bca1bea0034184b7b79472ff6b7ce5, State: Running, Role: LEADER
I20260809 06:48:59.848346 16855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.857023 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bca1bea0034184b7b79472ff6b7ce5. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.857466 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:59.859014 16900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:48:59.863044 16722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } attempt: 5
I20260809 06:48:59.863817 16900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 06:48:59.870324 16769 tablet.cc:2369] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 06:48:59 dist-test-slave-jpj1 krb5kdc[16499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
W20260809 06:48:59.879011 16551 tablet.cc:2369] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:48:59.880555 16732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.863625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:44467 (local address 127.0.0.1:55734)
0809 06:48:59.863973 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0809 06:48:59.863993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:48:59.864101 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:48:59.864982 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:48:59.864988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:48:59.865507 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:48:59.865722 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.865735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.867233 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.867239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.868694 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:48:59.868710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.869913 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.869919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:48:59.870129 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.870816 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:48:59.870839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:48:59.873662 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:48:59.877655 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.877661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.877665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.878170 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.878603 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:48:59.878608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:48:59.878612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:48:59.878769 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:48:59.879495 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:48:59.879500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:48:59.879730 (+ 230us) client_negotiation.cc:769] Sending connection context
0809 06:48:59.879875 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 06:48:59.880157 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260809 06:48:59.881654 16843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:48:59.864306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.27.209.252:44467)
0809 06:48:59.864471 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:48:59.864477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:48:59.864496 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:48:59.864548 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:48:59.864553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:48:59.864626 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:48:59.864724 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:48:59.865901 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.867031 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.868865 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:48:59.869601 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:48:59.869717 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:48:59.873894 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:48:59.873903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:48:59.873907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:48:59.873945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:48:59.877504 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.878325 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.878330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.878334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.878477 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:48:59.878895 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:48:59.878900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:48:59.878903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:48:59.879280 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:48:59.879433 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 06:48:59.881050 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0809 06:48:59.881304 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:48:59.887303 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: Bootstrap starting.
I20260809 06:48:59.891656 16722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: Neither blocks nor log segments found. Creating new log.
I20260809 06:48:59.893039 16722 log.cc:824] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: Log is configured to *not* fsync() on all Append() calls
I20260809 06:48:59.897854 16583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [term 1 FOLLOWER]: Refusing update from remote peer 23bca1bea0034184b7b79472ff6b7ce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:48:59.898938 16898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:48:59.905030 16722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: No bootstrap required, opened a new log
I20260809 06:48:59.916581 16722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.917124 16722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:48:59.917304 16722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617f92beb5b54a5cb28341e328e49978, State: Initialized, Role: FOLLOWER
I20260809 06:48:59.918040 16722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } }
I20260809 06:48:59.924997 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.927172 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:48:59.927042 16722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:48:59.927093 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23bca1bea0034184b7b79472ff6b7ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617f92beb5b54a5cb28341e328e49978" is_pre_election: true
I20260809 06:48:59.927834 16686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:48:59.931614 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23bca1bea0034184b7b79472ff6b7ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "617f92beb5b54a5cb28341e328e49978"
I20260809 06:48:59.932984 16686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [term 1 FOLLOWER]: Refusing update from remote peer 23bca1bea0034184b7b79472ff6b7ce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:48:59.933846 16898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:48:59.950546 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bca1bea0034184b7b79472ff6b7ce5. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.950901 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:48:59.955449 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bca1bea0034184b7b79472ff6b7ce5. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.955816 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:48:59.958026 16902 mvcc.cc:205] Tried to move back new op lower bound from 7316513341007949824 to 7316513340824100864. Current Snapshot: MvccSnapshot[applied={T|T < 7316513341007949824}]
I20260809 06:48:59.959940 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.960182 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:48:59.963205 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.963521 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:59.965207 16900 catalog_manager.cc:1347] Generated new cluster ID: c8c866fd23a4414c8da5ec60fd9d213f
I20260809 06:48:59.965306 16900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:48:59.979461 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.979746 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:48:59.985882 16900 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:48:59.986290 16911 mvcc.cc:205] Tried to move back new op lower bound from 7316513341007949824 to 7316513340824100864. Current Snapshot: MvccSnapshot[applied={T|T < 7316513341007949824}]
I20260809 06:48:59.987865 16900 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:48:59.991940 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23bca1bea0034184b7b79472ff6b7ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43d1356cd97c420bbeb10d6e113cb1b6" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38147 } } peers { permanent_uuid: "617f92beb5b54a5cb28341e328e49978" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 43687 } } peers { permanent_uuid: "23bca1bea0034184b7b79472ff6b7ce5" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 44467 } } }
I20260809 06:48:59.992224 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:49:00.021670 16900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5: Generated new TSK 0
I20260809 06:49:00.024295 16900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:49:00.036779 16940 catalog_manager.cc:1259] Loaded cluster ID: c8c866fd23a4414c8da5ec60fd9d213f
I20260809 06:49:00.036993 16940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: loading cluster ID for follower catalog manager: success
I20260809 06:49:00.040616 16940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: acquiring CA information for follower catalog manager: success
I20260809 06:49:00.043037 16940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 617f92beb5b54a5cb28341e328e49978: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:49:00.828090 16894 catalog_manager.cc:1259] Loaded cluster ID: c8c866fd23a4414c8da5ec60fd9d213f
I20260809 06:49:00.828190 16894 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: loading cluster ID for follower catalog manager: success
I20260809 06:49:00.830667 16894 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: acquiring CA information for follower catalog manager: success
I20260809 06:49:00.832453 16894 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43d1356cd97c420bbeb10d6e113cb1b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:49:01.177412 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 16520
I20260809 06:49:01.178545 16611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:01.287751 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 16520
I20260809 06:49:01.313652 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 16628
I20260809 06:49:01.314738 16714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:01.420442 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 16628
I20260809 06:49:01.444890 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:01.444113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38147 (local address 127.0.0.1:39314)
0809 06:49:01.444577 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:01.444704 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":260,"threads_started":1}
W20260809 06:49:01.445195 16762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.254:38147: connect: Connection refused (error 111)
I20260809 06:49:01.445797 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 16739
I20260809 06:49:01.446617 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:01.446266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38147 (local address 127.0.0.1:39322)
0809 06:49:01.446407 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:01.446490 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:38147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 06:49:01.446846 16831 generic_service.cc:196] Checking for leaks (request via RPC)
W20260809 06:49:01.446820 16762 consensus_peers.cc:487] T 00000000000000000000000000000000 P 23bca1bea0034184b7b79472ff6b7ce5 -> Peer 43d1356cd97c420bbeb10d6e113cb1b6 (127.27.209.254:38147): Couldn't send request to peer 43d1356cd97c420bbeb10d6e113cb1b6. Status: Network error: Client connection negotiation failed: client connection to 127.27.209.254:38147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 06:49:01.552713 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 16739
2026-08-09T06:49:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4802 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:49:01 dist-test-slave-jpj1 krb5kdc[16959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:49:01 dist-test-slave-jpj1 krb5kdc[16959](info): set up 2 sockets
Aug 09 06:49:01 dist-test-slave-jpj1 krb5kdc[16959](info): commencing operation
krb5kdc: starting...
W20260809 06:49:02.756453 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:49:02 dist-test-slave-jpj1 krb5kdc[16959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:49:03.015954 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37379
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:03.347121 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:03.347409 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:03.347568 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:03.354738 16979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:49:03.354851 16979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:03.354925 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:03.355014 16979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:49:03.355077 16979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:49:03.364761 16979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:37379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:37379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:03.366003 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:03.367290 16979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:03.378808 16990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:03.378836 16986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:03.379218 16987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:03.380034 16979 server_base.cc:1034] running on GCE node
I20260809 06:49:03.380724 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:03.381939 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:03.383138 16979 hybrid_clock.cc:648] HybridClock initialized: now 1786258143383088 us; error 42 us; skew 500 ppm
I20260809 06:49:03.383462 16979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:03 dist-test-slave-jpj1 krb5kdc[16959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:03.386425 16979 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:49:03.387809 16979 webserver.cc:460] Webserver started at http://127.27.209.254:34401/ using document root <none> and password file <none>
I20260809 06:49:03.388474 16979 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:03.388578 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:03.388833 16979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:03.391670 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "13279a0a268348eba2dfe712d59d2c33"
format_stamp: "Formatted at 2026-08-09 06:49:03 on dist-test-slave-jpj1"
server_key: "7c9407f20010466deec9d132172c075a"
server_key_iv: "06d690903093486988df08446d068322"
server_key_version: "encryptionkey@0"
I20260809 06:49:03.392390 16979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "13279a0a268348eba2dfe712d59d2c33"
format_stamp: "Formatted at 2026-08-09 06:49:03 on dist-test-slave-jpj1"
server_key: "7c9407f20010466deec9d132172c075a"
server_key_iv: "06d690903093486988df08446d068322"
server_key_version: "encryptionkey@0"
I20260809 06:49:03.397323 16979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:49:03.400672 17000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:03.401994 16979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:49:03.402141 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "13279a0a268348eba2dfe712d59d2c33"
format_stamp: "Formatted at 2026-08-09 06:49:03 on dist-test-slave-jpj1"
server_key: "7c9407f20010466deec9d132172c075a"
server_key_iv: "06d690903093486988df08446d068322"
server_key_version: "encryptionkey@0"
I20260809 06:49:03.402343 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:03.420193 16979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:03.423893 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:03.424144 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:03.440917 16979 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:37379
I20260809 06:49:03.440935 17084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:37379 every 8 connection(s)
I20260809 06:49:03.442203 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:49:03.445776 17085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:03.449601 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 16979
I20260809 06:49:03.449834 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:49:03.450650 28487 external_mini_cluster.cc:1428] Setting key 56be2dd82a3a6c47c4e3fb183d062d70
I20260809 06:49:03.458637 17085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33: Bootstrap starting.
I20260809 06:49:03.462373 17085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:03.463721 17085 log.cc:824] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33: Log is configured to *not* fsync() on all Append() calls
Aug 09 06:49:03 dist-test-slave-jpj1 krb5kdc[16959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:03.466953 17085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33: No bootstrap required, opened a new log
I20260809 06:49:03.473677 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:03.452439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.27.209.254:37379)
0809 06:49:03.453158 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:03.453169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:03.453214 (+ 45us) server_negotiation.cc:407] Connection header received
0809 06:49:03.454792 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:03.454815 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:03.455188 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:03.455481 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:03.457200 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:03.458424 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:03.460324 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:03.461003 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:03.461126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:03.466371 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:03.466378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:03.466394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:03.466428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:03.469489 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:03.470463 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:03.470469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:03.470475 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:03.470647 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:03.471030 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:03.471033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:03.471035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:03.471596 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:03.471832 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:03.472257 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 06:49:03.472619 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":333,"threads_started":1}
I20260809 06:49:03.475687 17085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13279a0a268348eba2dfe712d59d2c33" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37379 } }
I20260809 06:49:03.476053 17085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:03.476161 17085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13279a0a268348eba2dfe712d59d2c33, State: Initialized, Role: FOLLOWER
I20260809 06:49:03.476735 17085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13279a0a268348eba2dfe712d59d2c33" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37379 } }
I20260809 06:49:03.476969 17085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:49:03.477115 17085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:49:03.477274 17085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:03.477388 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:49:03.480366 17085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13279a0a268348eba2dfe712d59d2c33" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37379 } }
I20260809 06:49:03.480891 17085 leader_election.cc:302] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13279a0a268348eba2dfe712d59d2c33; no voters:
I20260809 06:49:03.481530 17085 leader_election.cc:288] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:49:03.481635 17091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:03.482355 17091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [term 1 LEADER]: Becoming Leader. State: Replica: 13279a0a268348eba2dfe712d59d2c33, State: Running, Role: LEADER
I20260809 06:49:03.482899 17091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13279a0a268348eba2dfe712d59d2c33" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37379 } }
I20260809 06:49:03.483865 17085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:49:03.485695 17092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13279a0a268348eba2dfe712d59d2c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13279a0a268348eba2dfe712d59d2c33" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37379 } } }
I20260809 06:49:03.486052 17092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:03.487092 17093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13279a0a268348eba2dfe712d59d2c33. Latest consensus state: current_term: 1 leader_uuid: "13279a0a268348eba2dfe712d59d2c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13279a0a268348eba2dfe712d59d2c33" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 37379 } } }
I20260809 06:49:03.487848 17100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:49:03.487953 17093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33 [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:03.492069 17100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:49:03.500909 17100 catalog_manager.cc:1347] Generated new cluster ID: a39365d9ef5b421380e7d410c5a1db19
I20260809 06:49:03.501021 17100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:49:03.524369 17100 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:49:03.525646 17100 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:49:03.538143 17100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13279a0a268348eba2dfe712d59d2c33: Generated new TSK 0
I20260809 06:49:03.538944 17100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:49:03.590317 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 16979
I20260809 06:49:03.591483 17078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:03.685885 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 16979
2026-08-09T06:49:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2129 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:49:03 dist-test-slave-jpj1 krb5kdc[17131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:49:03 dist-test-slave-jpj1 krb5kdc[17131](info): set up 2 sockets
Aug 09 06:49:03 dist-test-slave-jpj1 krb5kdc[17131](info): commencing operation
krb5kdc: starting...
W20260809 06:49:04.885533 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:49:05 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:49:05.142798 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:32857
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:05.470666 17156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:05.470954 17156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:05.471086 17156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:05.478252 17156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:49:05.478360 17156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:05.478436 17156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:05.478511 17156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:49:05.478569 17156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:49:05.488511 17156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:05.489877 17156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:05.491168 17156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:05.502489 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:05.502532 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:05.502558 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:05.503602 17156 server_base.cc:1034] running on GCE node
I20260809 06:49:05.504336 17156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:05.505662 17156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:05.506875 17156 hybrid_clock.cc:648] HybridClock initialized: now 1786258145506823 us; error 39 us; skew 500 ppm
I20260809 06:49:05.507200 17156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:05 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258145, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:05.510262 17156 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:49:05.511754 17156 webserver.cc:460] Webserver started at http://127.27.209.254:33179/ using document root <none> and password file <none>
I20260809 06:49:05.512418 17156 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:05.512543 17156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:05.512850 17156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:05.515820 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "3e6bb7897a2c40979e8450cdc191d23f"
format_stamp: "Formatted at 2026-08-09 06:49:05 on dist-test-slave-jpj1"
server_key: "e8aefa20cc29412a32db078ca4d86f81"
server_key_iv: "54e0a58da15eb1195f133271e652c8ef"
server_key_version: "encryptionkey@0"
I20260809 06:49:05.516566 17156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "3e6bb7897a2c40979e8450cdc191d23f"
format_stamp: "Formatted at 2026-08-09 06:49:05 on dist-test-slave-jpj1"
server_key: "e8aefa20cc29412a32db078ca4d86f81"
server_key_iv: "54e0a58da15eb1195f133271e652c8ef"
server_key_version: "encryptionkey@0"
I20260809 06:49:05.521656 17156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:49:05.524997 17178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:05.526314 17156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 06:49:05.526489 17156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "3e6bb7897a2c40979e8450cdc191d23f"
format_stamp: "Formatted at 2026-08-09 06:49:05 on dist-test-slave-jpj1"
server_key: "e8aefa20cc29412a32db078ca4d86f81"
server_key_iv: "54e0a58da15eb1195f133271e652c8ef"
server_key_version: "encryptionkey@0"
I20260809 06:49:05.526659 17156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:05.542932 17156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:05.546595 17156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:05.546852 17156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:05.564211 17156 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:32857
I20260809 06:49:05.564234 17250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:32857 every 8 connection(s)
I20260809 06:49:05.565500 17156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:49:05.566066 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 17156
I20260809 06:49:05.566282 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:49:05.567068 28487 external_mini_cluster.cc:1428] Setting key c284d00ae6036b0018f12da68ef245ab
I20260809 06:49:05.569694 17251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:49:05 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:05.582336 17251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap starting.
I20260809 06:49:05.586345 17251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:05.587505 17251 log.cc:824] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:05.590601 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:05.568943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.27.209.254:32857)
0809 06:49:05.569677 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:05.569693 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:05.569735 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:49:05.571326 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:05.571348 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:05.571744 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:05.572060 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:05.573800 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:05.575104 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:05.576979 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:05.577626 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:05.577770 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:05.582376 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:05.582387 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:05.582406 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:05.582461 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:05.586261 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:05.586919 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:05.586928 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:05.586936 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:05.587138 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:05.587506 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:05.587511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:05.587514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:05.588232 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:05.588494 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:05.588914 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 06:49:05.589340 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":327,"threads_started":1}
I20260809 06:49:05.590910 17251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: No bootstrap required, opened a new log
I20260809 06:49:05.598430 17251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:05.598820 17251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:05.598920 17251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6bb7897a2c40979e8450cdc191d23f, State: Initialized, Role: FOLLOWER
I20260809 06:49:05.599480 17251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:05.599762 17251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:49:05.599896 17251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:49:05.600051 17251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:05.602046 17251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:05.602505 17251 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6bb7897a2c40979e8450cdc191d23f; no voters:
I20260809 06:49:05.603073 17251 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:49:05.603245 17259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:05.604256 17259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 LEADER]: Becoming Leader. State: Replica: 3e6bb7897a2c40979e8450cdc191d23f, State: Running, Role: LEADER
I20260809 06:49:05.604775 17259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:05.605077 17251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:49:05.607353 17262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6bb7897a2c40979e8450cdc191d23f. Latest consensus state: current_term: 1 leader_uuid: "3e6bb7897a2c40979e8450cdc191d23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } } }
I20260809 06:49:05.607749 17262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:05.607704 17261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e6bb7897a2c40979e8450cdc191d23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } } }
I20260809 06:49:05.608052 17261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:05.612483 17274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:49:05.617974 17274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:49:05.626719 17274 catalog_manager.cc:1347] Generated new cluster ID: 5aae535af97448fea313523d1b05f320
I20260809 06:49:05.626816 17274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:49:05.639528 17274 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:49:05.640765 17274 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:49:05.650039 17274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Generated new TSK 0
I20260809 06:49:05.650882 17274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:49:05.717628 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:32857
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:06.050218 17296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:06.050535 17296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:06.050678 17296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:06.057871 17296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:06.057996 17296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:06.058152 17296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:49:06.067780 17296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:06.069077 17296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:06.070456 17296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:06.083436 17302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:06.083472 17303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:06.083527 17305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:06.084160 17296 server_base.cc:1034] running on GCE node
I20260809 06:49:06.084820 17296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:06.085707 17296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:06.086910 17296 hybrid_clock.cc:648] HybridClock initialized: now 1786258146086862 us; error 33 us; skew 500 ppm
I20260809 06:49:06.087203 17296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:06 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258146, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:06.090387 17296 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:49:06.091831 17296 webserver.cc:460] Webserver started at http://127.27.209.193:38933/ using document root <none> and password file <none>
I20260809 06:49:06.092533 17296 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:06.092651 17296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:06.092948 17296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:06.095883 17296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "4e9b178312484c11b3f00ab19fee4807"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "e68a5e59e8df38999cbce7113781b745"
server_key_iv: "cdad22dcb4e0702a7b0b02b32a3da741"
server_key_version: "encryptionkey@0"
I20260809 06:49:06.096652 17296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "4e9b178312484c11b3f00ab19fee4807"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "e68a5e59e8df38999cbce7113781b745"
server_key_iv: "cdad22dcb4e0702a7b0b02b32a3da741"
server_key_version: "encryptionkey@0"
I20260809 06:49:06.101709 17296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:49:06.105057 17317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:06.106364 17296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:49:06.106514 17296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "4e9b178312484c11b3f00ab19fee4807"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "e68a5e59e8df38999cbce7113781b745"
server_key_iv: "cdad22dcb4e0702a7b0b02b32a3da741"
server_key_version: "encryptionkey@0"
I20260809 06:49:06.106711 17296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:06.124137 17296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:06.127929 17296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:06.128187 17296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:06.129103 17296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:06.130470 17296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:06.130546 17296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:06.130671 17296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:06.130724 17296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:06.160199 17296 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:43779
I20260809 06:49:06.160238 17487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:43779 every 8 connection(s)
I20260809 06:49:06.161442 17296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:49:06.163663 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 17296
I20260809 06:49:06.163887 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:49:06.164674 28487 external_mini_cluster.cc:1428] Setting key cca07473c2f512b3b696cd3b1dab9d6f
Aug 09 06:49:06 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258146, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:06.181401 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:06.163995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:34749 (local address 127.27.209.254:32857)
0809 06:49:06.164490 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:06.164497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:06.165562 (+ 1065us) server_negotiation.cc:407] Connection header received
0809 06:49:06.167368 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:06.167373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:06.167450 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:06.167589 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:06.169031 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.169996 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.171470 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.172039 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.172170 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:06.175712 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:06.175717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:06.175721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:06.175755 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:06.178348 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:06.179065 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:06.179071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:06.179075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:06.179220 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:06.179686 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:06.179691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:06.179695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:06.180040 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:06.180178 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:06.180832 (+ 654us) server_negotiation.cc:299] Negotiation successful
0809 06:49:06.181045 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260809 06:49:06.182104 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:06.164587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:34749)
0809 06:49:06.165403 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:06.165456 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:06.167102 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:06.167790 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:06.167799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:06.168295 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:06.168865 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:06.168883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.170112 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.170115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:06.171324 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:06.171336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.172239 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.172245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:06.172440 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:06.173055 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:06.173081 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:06.175470 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:06.178514 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:06.178521 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:06.178533 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:06.178948 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:06.179351 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:06.179354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:06.179357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:06.179578 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:06.180143 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:06.180148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:06.180458 (+ 310us) client_negotiation.cc:769] Sending connection context
0809 06:49:06.180680 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 06:49:06.181036 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":322,"threads_started":1}
I20260809 06:49:06.183715 17490 heartbeater.cc:347] Connected to a master server at 127.27.209.254:32857
I20260809 06:49:06.184003 17490 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:06.184736 17490 heartbeater.cc:511] Master 127.27.209.254:32857 requested a full tablet report, sending...
I20260809 06:49:06.186990 17202 ts_manager.cc:194] Registered new tserver with Master: 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193:43779)
I20260809 06:49:06.188544 17202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:34749
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:49:06.222338 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:32857
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:06.543337 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:06.543646 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:06.543759 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:06.550740 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:06.550841 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:06.550951 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:49:06.560334 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:06.561523 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:06.562822 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:06.575347 17507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:06.575381 17508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:06.576282 17510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:06.577303 17499 server_base.cc:1034] running on GCE node
I20260809 06:49:06.577975 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:06.579051 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:06.580272 17499 hybrid_clock.cc:648] HybridClock initialized: now 1786258146580218 us; error 38 us; skew 500 ppm
I20260809 06:49:06.580564 17499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:06 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258146, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:06.583501 17499 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:49:06.584812 17499 webserver.cc:460] Webserver started at http://127.27.209.194:36865/ using document root <none> and password file <none>
I20260809 06:49:06.585489 17499 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:06.585603 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:06.585879 17499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:06.588824 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "578def645c8f4b87967295d852a2149f"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "7ca8fb2ea5c8aabf3c8ca92ee6f4c7ee"
server_key_iv: "0115a15bee69624b0559fe9a0f1e4de5"
server_key_version: "encryptionkey@0"
I20260809 06:49:06.589560 17499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "578def645c8f4b87967295d852a2149f"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "7ca8fb2ea5c8aabf3c8ca92ee6f4c7ee"
server_key_iv: "0115a15bee69624b0559fe9a0f1e4de5"
server_key_version: "encryptionkey@0"
I20260809 06:49:06.594563 17499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:49:06.597921 17518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:06.599193 17499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:06.599349 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "578def645c8f4b87967295d852a2149f"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "7ca8fb2ea5c8aabf3c8ca92ee6f4c7ee"
server_key_iv: "0115a15bee69624b0559fe9a0f1e4de5"
server_key_version: "encryptionkey@0"
I20260809 06:49:06.599568 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:06.616395 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:06.620465 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:06.620733 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:06.621659 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:06.622984 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:06.623057 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:06.623175 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:06.623222 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:06.651268 17499 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:40675
I20260809 06:49:06.651299 17683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:40675 every 8 connection(s)
I20260809 06:49:06.652477 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:49:06.654459 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 17499
I20260809 06:49:06.654657 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:49:06.655256 28487 external_mini_cluster.cc:1428] Setting key 5682d1048fe2809516a68304ccdeedc4
Aug 09 06:49:06 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258146, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:06.676069 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:06.654912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:35833 (local address 127.27.209.254:32857)
0809 06:49:06.655075 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:06.655080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:06.656610 (+ 1530us) server_negotiation.cc:407] Connection header received
0809 06:49:06.658627 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:06.658632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:06.658713 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:06.658795 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:06.660308 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.661015 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.662553 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.663007 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.663124 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:06.670735 (+ 7611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:06.670741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:06.670745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:06.670782 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:06.673179 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:06.673871 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:06.673875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:06.673878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:06.674002 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:06.674408 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:06.674411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:06.674413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:06.674667 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:06.674798 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:06.675483 (+ 685us) server_negotiation.cc:299] Negotiation successful
0809 06:49:06.675736 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 06:49:06.677073 17687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:06.655620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:35833)
0809 06:49:06.656447 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:06.656497 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:06.658373 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:06.659031 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:06.659039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:06.659525 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:06.660135 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:06.660154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.661167 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.661171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:06.662412 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:06.662425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:06.663237 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:06.663245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:06.663471 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:06.668039 (+ 4568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:06.668068 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:06.670513 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:06.673313 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:06.673322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:06.673340 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:06.673752 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:06.674105 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:06.674109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:06.674113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:06.674294 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:06.674795 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:06.674803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:06.675123 (+ 320us) client_negotiation.cc:769] Sending connection context
0809 06:49:06.675362 (+ 239us) client_negotiation.cc:240] Negotiation successful
0809 06:49:06.675914 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":318,"threads_started":1}
I20260809 06:49:06.678421 17684 heartbeater.cc:347] Connected to a master server at 127.27.209.254:32857
I20260809 06:49:06.678781 17684 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:06.679719 17684 heartbeater.cc:511] Master 127.27.209.254:32857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
I20260809 06:49:06.681427 17202 ts_manager.cc:194] Registered new tserver with Master: 578def645c8f4b87967295d852a2149f (127.27.209.194:40675)
I20260809 06:49:06.682332 17202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:35833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:49:06.714884 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:32857
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:07.039009 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:07.039304 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:07.039433 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:07.046732 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:07.046873 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:07.047041 17690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:49:07.056818 17690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:07.058136 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:07.059514 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:07.072500 17699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:07.072463 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:07.072540 17701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:07.073657 17690 server_base.cc:1034] running on GCE node
I20260809 06:49:07.074256 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:07.075188 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:07.076416 17690 hybrid_clock.cc:648] HybridClock initialized: now 1786258147076355 us; error 47 us; skew 500 ppm
I20260809 06:49:07.076766 17690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:07 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258147, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:07.080024 17690 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:49:07.081439 17690 webserver.cc:460] Webserver started at http://127.27.209.195:39115/ using document root <none> and password file <none>
I20260809 06:49:07.082099 17690 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:07.082201 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:07.082460 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:07.085407 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "18ff6bb28e314a24b793eb4167f6cc16"
format_stamp: "Formatted at 2026-08-09 06:49:07 on dist-test-slave-jpj1"
server_key: "6408f9ffca3cbb79223af66ecae6e307"
server_key_iv: "0d0b4f34fee6a6bdae5f66a4ee2b1f21"
server_key_version: "encryptionkey@0"
I20260809 06:49:07.086159 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "18ff6bb28e314a24b793eb4167f6cc16"
format_stamp: "Formatted at 2026-08-09 06:49:07 on dist-test-slave-jpj1"
server_key: "6408f9ffca3cbb79223af66ecae6e307"
server_key_iv: "0d0b4f34fee6a6bdae5f66a4ee2b1f21"
server_key_version: "encryptionkey@0"
I20260809 06:49:07.091423 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:49:07.095010 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:07.096493 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:49:07.096630 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "18ff6bb28e314a24b793eb4167f6cc16"
format_stamp: "Formatted at 2026-08-09 06:49:07 on dist-test-slave-jpj1"
server_key: "6408f9ffca3cbb79223af66ecae6e307"
server_key_iv: "0d0b4f34fee6a6bdae5f66a4ee2b1f21"
server_key_version: "encryptionkey@0"
I20260809 06:49:07.096813 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:07.122462 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:07.126406 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:07.126643 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:07.127561 17690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:07.128882 17690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:07.128957 17690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:07.129047 17690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:07.129091 17690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:07.158594 17690 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:45389
I20260809 06:49:07.158639 17886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:45389 every 8 connection(s)
I20260809 06:49:07.159986 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:49:07.169229 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 17690
I20260809 06:49:07.169399 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:49:07.169854 28487 external_mini_cluster.cc:1428] Setting key 4e22d3d5e01691530810dc44e0ccc92d
Aug 09 06:49:07 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258147, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:07.178997 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.162594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:33023 (local address 127.27.209.254:32857)
0809 06:49:07.162774 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.162780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.164154 (+ 1374us) server_negotiation.cc:407] Connection header received
0809 06:49:07.165906 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.165912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.166003 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.166091 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:07.167518 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.168291 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.169860 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.170324 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.170458 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.174055 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:07.174060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:07.174063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:07.174095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:07.176437 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.177073 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.177077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.177079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.177200 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.177525 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.177528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.177530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.177791 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:07.177905 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.178581 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.178741 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:49:07.179975 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.163230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:33023)
0809 06:49:07.163998 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:07.164041 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:07.165638 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:07.166336 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:07.166345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:07.166813 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:07.167343 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.167361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.168444 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.168448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.169720 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.169737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.170534 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.170541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.170854 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.171555 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:07.171586 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:07.173876 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:07.176559 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:07.176566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:07.176578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:07.176983 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:07.177275 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:07.177278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:07.177280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:07.177450 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:07.177913 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:07.177922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:07.178255 (+ 333us) client_negotiation.cc:769] Sending connection context
0809 06:49:07.178479 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 06:49:07.178849 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":331,"threads_started":1}
I20260809 06:49:07.181222 17887 heartbeater.cc:347] Connected to a master server at 127.27.209.254:32857
I20260809 06:49:07.181514 17887 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:07.182219 17887 heartbeater.cc:511] Master 127.27.209.254:32857 requested a full tablet report, sending...
I20260809 06:49:07.183679 17202 ts_manager.cc:194] Registered new tserver with Master: 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195:45389)
I20260809 06:49:07.184595 17202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:33023
I20260809 06:49:07.185212 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:49:07.191355 17490 heartbeater.cc:503] Master 127.27.209.254:32857 was elected leader, sending a full tablet report...
I20260809 06:49:07.201941 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.190245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.27.209.254:32857)
0809 06:49:07.190415 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.190421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.190848 (+ 427us) server_negotiation.cc:407] Connection header received
0809 06:49:07.190967 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.190970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.191026 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.191132 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:07.192603 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.193393 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.194686 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.195064 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.195152 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.197393 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:07.197398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:07.197401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:07.197432 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:07.199724 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.200330 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.200334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.200336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.200461 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.200774 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.200777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.200779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.201068 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:07.201283 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.201526 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.201684 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:49:07.206036 17202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:49:07.213032 17202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:49:07.236523 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.224922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.27.209.194:40675)
0809 06:49:07.226501 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.226512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.228881 (+ 2369us) server_negotiation.cc:407] Connection header received
0809 06:49:07.229207 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.229213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.229467 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.229700 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:07.230323 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.231810 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.234066 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.235197 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.235363 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.235521 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.236077 (+ 556us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.236251 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":2912,"threads_started":1}
I20260809 06:49:07.236515 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.224540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:43779 (local address 127.0.0.1:42666)
0809 06:49:07.227009 (+ 2469us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:07.227044 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:07.227203 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:07.228681 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:07.228689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:07.228737 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:07.229196 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.229207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.231083 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.231087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.233628 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.233643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.235204 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.235210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.235615 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.235676 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 06:49:07.235812 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 06:49:07.235999 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2257,"mutex_wait_us":56,"thread_start_us":2120,"threads_started":1}
I20260809 06:49:07.236935 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.225378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.27.209.193:43779)
0809 06:49:07.227767 (+ 2389us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.227774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.227802 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:49:07.227900 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.227908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.228163 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.228371 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:07.229362 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.230916 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.233813 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.234995 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.235185 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.235356 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.236466 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.236626 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2230,"thread_start_us":2028,"threads_started":1}
I20260809 06:49:07.237250 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.226438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:45389 (local address 127.0.0.1:52446)
0809 06:49:07.228911 (+ 2473us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:07.228925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:07.229048 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:07.230134 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:07.230139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:07.230168 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:07.230555 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.230567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.232459 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.232464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.234427 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.234442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.236470 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.236476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.236634 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.236665 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:49:07.236749 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 06:49:07.236871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2329,"mutex_wait_us":3,"thread_start_us":2183,"threads_started":1}
I20260809 06:49:07.237592 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.224873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40675 (local address 127.0.0.1:38896)
0809 06:49:07.228764 (+ 3891us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:07.228780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:07.228979 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:07.229707 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:07.229712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:07.229736 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:07.230093 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.230106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.231981 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.231986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.233912 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.233926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.235686 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.235692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.235878 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.235913 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 06:49:07.237145 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0809 06:49:07.237269 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2722,"mutex_wait_us":62,"thread_start_us":2764,"threads_started":1}
I20260809 06:49:07.238004 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.227067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.27.209.195:45389)
0809 06:49:07.228465 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.228474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.229020 (+ 546us) server_negotiation.cc:407] Connection header received
0809 06:49:07.229689 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.229696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.229935 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.230130 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:07.230755 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.232303 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.234591 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.235945 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.236135 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.236303 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.237533 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.237680 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":1071,"threads_started":1}
I20260809 06:49:07.240913 17591 tablet_service.cc:1467] Processing CreateTablet for tablet f817ea58d7174f2596604786f49a099b (DEFAULT_TABLE table=test-table [id=41880d8997764f1390186abd98d38a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:07.240934 17392 tablet_service.cc:1467] Processing CreateTablet for tablet f817ea58d7174f2596604786f49a099b (DEFAULT_TABLE table=test-table [id=41880d8997764f1390186abd98d38a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:07.241207 17791 tablet_service.cc:1467] Processing CreateTablet for tablet f817ea58d7174f2596604786f49a099b (DEFAULT_TABLE table=test-table [id=41880d8997764f1390186abd98d38a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:07.242228 17591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f817ea58d7174f2596604786f49a099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:07.242406 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f817ea58d7174f2596604786f49a099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:07.242406 17392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f817ea58d7174f2596604786f49a099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:07.254354 17919 tablet_bootstrap.cc:492] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Bootstrap starting.
I20260809 06:49:07.256600 17921 tablet_bootstrap.cc:492] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Bootstrap starting.
I20260809 06:49:07.257592 17919 tablet_bootstrap.cc:654] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:07.257761 17923 tablet_bootstrap.cc:492] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Bootstrap starting.
I20260809 06:49:07.258920 17919 log.cc:824] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:07.261116 17923 tablet_bootstrap.cc:654] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:07.262415 17923 log.cc:824] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:07.262444 17921 tablet_bootstrap.cc:654] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:07.262974 17919 tablet_bootstrap.cc:492] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: No bootstrap required, opened a new log
I20260809 06:49:07.263348 17919 ts_tablet_manager.cc:1397] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260809 06:49:07.263602 17921 log.cc:824] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:07.266093 17923 tablet_bootstrap.cc:492] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: No bootstrap required, opened a new log
I20260809 06:49:07.266366 17923 ts_tablet_manager.cc:1397] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:49:07.266708 17921 tablet_bootstrap.cc:492] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: No bootstrap required, opened a new log
I20260809 06:49:07.266953 17921 ts_tablet_manager.cc:1397] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260809 06:49:07.270787 17919 raft_consensus.cc:348] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.271116 17919 raft_consensus.cc:374] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:07.271184 17919 raft_consensus.cc:729] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578def645c8f4b87967295d852a2149f, State: Initialized, Role: FOLLOWER
I20260809 06:49:07.271764 17919 consensus_queue.cc:260] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.272841 17684 heartbeater.cc:503] Master 127.27.209.254:32857 was elected leader, sending a full tablet report...
I20260809 06:49:07.273522 17919 ts_tablet_manager.cc:1428] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260809 06:49:07.276413 17923 raft_consensus.cc:348] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.276898 17923 raft_consensus.cc:374] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:07.277005 17923 raft_consensus.cc:729] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ff6bb28e314a24b793eb4167f6cc16, State: Initialized, Role: FOLLOWER
I20260809 06:49:07.277171 17921 raft_consensus.cc:348] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.277598 17921 raft_consensus.cc:374] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:07.277707 17921 raft_consensus.cc:729] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b178312484c11b3f00ab19fee4807, State: Initialized, Role: FOLLOWER
I20260809 06:49:07.277588 17923 consensus_queue.cc:260] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.278297 17921 consensus_queue.cc:260] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.278759 17887 heartbeater.cc:503] Master 127.27.209.254:32857 was elected leader, sending a full tablet report...
I20260809 06:49:07.279547 17923 ts_tablet_manager.cc:1428] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 06:49:07.280766 17921 ts_tablet_manager.cc:1428] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260809 06:49:07.404299 17686 tablet.cc:2369] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:49:07.411619 17890 tablet.cc:2369] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:49:07.413642 17492 tablet.cc:2369] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:07.459476 17930 raft_consensus.cc:482] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:49:07.459816 17930 raft_consensus.cc:504] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.461480 17930 leader_election.cc:288] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193:43779), 578def645c8f4b87967295d852a2149f (127.27.209.194:40675)
I20260809 06:49:07.468272 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.461992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:59695 (local address 127.27.209.193:43779)
0809 06:49:07.462176 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.462181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.462323 (+ 142us) server_negotiation.cc:407] Connection header received
0809 06:49:07.462504 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.462510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.462608 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.462717 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:07.463202 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.464192 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.466264 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.467275 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.467433 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.467511 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.467916 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.468054 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:49:07.468282 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.462048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:43779 (local address 127.27.209.195:59695)
0809 06:49:07.462224 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:07.462243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:07.462371 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:07.462762 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:07.462771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:07.462805 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:07.463021 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.463033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.464330 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.464334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.466094 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.466110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.467459 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.467466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.467738 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.467767 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 06:49:07.467862 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 06:49:07.467963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260809 06:49:07.468840 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.461948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:33853 (local address 127.27.209.194:40675)
0809 06:49:07.462163 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.462169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.463156 (+ 987us) server_negotiation.cc:407] Connection header received
0809 06:49:07.463292 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.463297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.463375 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.463481 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:07.463989 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.464807 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.466826 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.467906 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.468055 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.468121 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.468507 (+ 386us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.468649 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:49:07.469522 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.462066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40675 (local address 127.27.209.195:33853)
0809 06:49:07.463010 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:07.463042 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:07.463162 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:07.463497 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:07.463500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:07.463520 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:07.463832 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.463840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.464951 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.464956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.466665 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:07.466678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.468111 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.468117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:07.468304 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.468338 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:49:07.469028 (+ 690us) client_negotiation.cc:240] Negotiation successful
0809 06:49:07.469175 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":135,"thread_start_us":381,"threads_started":1}
I20260809 06:49:07.469458 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f817ea58d7174f2596604786f49a099b" candidate_uuid: "18ff6bb28e314a24b793eb4167f6cc16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b178312484c11b3f00ab19fee4807" is_pre_election: true
I20260809 06:49:07.470109 17425 raft_consensus.cc:2454] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ff6bb28e314a24b793eb4167f6cc16 in term 0.
I20260809 06:49:07.470484 17615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f817ea58d7174f2596604786f49a099b" candidate_uuid: "18ff6bb28e314a24b793eb4167f6cc16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578def645c8f4b87967295d852a2149f" is_pre_election: true
I20260809 06:49:07.470858 17713 leader_election.cc:302] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ff6bb28e314a24b793eb4167f6cc16, 4e9b178312484c11b3f00ab19fee4807; no voters:
I20260809 06:49:07.471010 17615 raft_consensus.cc:2454] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18ff6bb28e314a24b793eb4167f6cc16 in term 0.
I20260809 06:49:07.471226 17930 raft_consensus.cc:2781] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:49:07.471349 17930 raft_consensus.cc:482] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:49:07.471463 17930 raft_consensus.cc:3037] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:07.473774 17930 raft_consensus.cc:504] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.474382 17930 leader_election.cc:288] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [CANDIDATE]: Term 1 election: Requested vote from peers 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193:43779), 578def645c8f4b87967295d852a2149f (127.27.209.194:40675)
I20260809 06:49:07.474833 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f817ea58d7174f2596604786f49a099b" candidate_uuid: "18ff6bb28e314a24b793eb4167f6cc16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b178312484c11b3f00ab19fee4807"
I20260809 06:49:07.475035 17425 raft_consensus.cc:3037] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:07.475008 17615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f817ea58d7174f2596604786f49a099b" candidate_uuid: "18ff6bb28e314a24b793eb4167f6cc16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578def645c8f4b87967295d852a2149f"
I20260809 06:49:07.475210 17615 raft_consensus.cc:3037] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:07.477502 17615 raft_consensus.cc:2454] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ff6bb28e314a24b793eb4167f6cc16 in term 1.
I20260809 06:49:07.477506 17425 raft_consensus.cc:2454] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18ff6bb28e314a24b793eb4167f6cc16 in term 1.
I20260809 06:49:07.477900 17713 leader_election.cc:302] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ff6bb28e314a24b793eb4167f6cc16, 4e9b178312484c11b3f00ab19fee4807; no voters:
I20260809 06:49:07.478205 17930 raft_consensus.cc:2781] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:07.478690 17930 raft_consensus.cc:686] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 LEADER]: Becoming Leader. State: Replica: 18ff6bb28e314a24b793eb4167f6cc16, State: Running, Role: LEADER
I20260809 06:49:07.479236 17930 consensus_queue.cc:237] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:07.483950 17201 catalog_manager.cc:5495] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195). New cstate: current_term: 1 leader_uuid: "18ff6bb28e314a24b793eb4167f6cc16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } health_report { overall_health: HEALTHY } } }
I20260809 06:49:07.531963 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.525291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.27.209.195:45389)
0809 06:49:07.525512 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.525518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.525540 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:49:07.525675 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.525680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.525803 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.525967 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:49:07.526377 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.527495 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.529206 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.529856 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.529974 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.530614 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:49:07.531312 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:49:07.531452 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.531585 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.531672 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:49:07.541582 17425 raft_consensus.cc:1261] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Refusing update from remote peer 18ff6bb28e314a24b793eb4167f6cc16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:07.541703 17615 raft_consensus.cc:1261] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Refusing update from remote peer 18ff6bb28e314a24b793eb4167f6cc16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:07.542729 17934 consensus_queue.cc:1035] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:07.543164 17930 consensus_queue.cc:1035] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:07.570789 17201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:07.571084 17201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:07.575138 17201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Sending DeleteTablet for 3 replicas of tablet f817ea58d7174f2596604786f49a099b
I20260809 06:49:07.576290 17392 tablet_service.cc:1514] Processing DeleteTablet for tablet f817ea58d7174f2596604786f49a099b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:07 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42666
I20260809 06:49:07.576443 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet f817ea58d7174f2596604786f49a099b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:07 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:52446
I20260809 06:49:07.576490 17591 tablet_service.cc:1514] Processing DeleteTablet for tablet f817ea58d7174f2596604786f49a099b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:07 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:38896
I20260809 06:49:07.577715 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 17296
I20260809 06:49:07.587704 17959 tablet_replica.cc:330] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: stopping tablet replica
I20260809 06:49:07.588156 17959 raft_consensus.cc:2229] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:49:07.589727 17961 tablet_replica.cc:330] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: stopping tablet replica
I20260809 06:49:07.590135 17961 raft_consensus.cc:2229] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:07.590585 17961 raft_consensus.cc:2258] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:07.588851 17959 raft_consensus.cc:2258] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:07.593405 17957 tablet_replica.cc:330] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: stopping tablet replica
I20260809 06:49:07.593807 17957 raft_consensus.cc:2229] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:07.595029 17959 ts_tablet_manager.cc:1905] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:07.595204 17957 raft_consensus.cc:2258] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:07.597373 17957 ts_tablet_manager.cc:1905] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:49:07 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:49:07.601131 17961 ts_tablet_manager.cc:1905] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:07.602717 17959 ts_tablet_manager.cc:1918] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:07.603012 17959 log.cc:1196] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/f817ea58d7174f2596604786f49a099b
I20260809 06:49:07.603561 17959 ts_tablet_manager.cc:1939] T f817ea58d7174f2596604786f49a099b P 18ff6bb28e314a24b793eb4167f6cc16: Deleting consensus metadata
I20260809 06:49:07.605489 17182 catalog_manager.cc:4841] TS 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195:45389): tablet f817ea58d7174f2596604786f49a099b (table test-table [id=41880d8997764f1390186abd98d38a24]) successfully deleted
I20260809 06:49:07.609788 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.578913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.27.209.193:43779)
0809 06:49:07.579084 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.579090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.586452 (+ 7362us) server_negotiation.cc:407] Connection header received
0809 06:49:07.586632 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.586638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.586731 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.586835 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:07.591641 (+ 4806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.592893 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.595113 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.595854 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.595954 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.601968 (+ 6014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:07.601984 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:07.602003 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:07.602049 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:07.606097 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.606942 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.606951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.606955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.607122 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.607595 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.607601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.607604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.608106 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:07.608464 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.609084 (+ 620us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.609360 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:49:07.610129 17961 ts_tablet_manager.cc:1918] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:07.610184 17957 ts_tablet_manager.cc:1918] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:07.610306 17961 log.cc:1196] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/f817ea58d7174f2596604786f49a099b
I20260809 06:49:07.610332 17957 log.cc:1196] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/f817ea58d7174f2596604786f49a099b
I20260809 06:49:07.610870 17961 ts_tablet_manager.cc:1939] T f817ea58d7174f2596604786f49a099b P 578def645c8f4b87967295d852a2149f: Deleting consensus metadata
I20260809 06:49:07.610918 17957 ts_tablet_manager.cc:1939] T f817ea58d7174f2596604786f49a099b P 4e9b178312484c11b3f00ab19fee4807: Deleting consensus metadata
I20260809 06:49:07.611851 17483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:07.612648 17183 catalog_manager.cc:4841] TS 578def645c8f4b87967295d852a2149f (127.27.209.194:40675): tablet f817ea58d7174f2596604786f49a099b (table test-table [id=41880d8997764f1390186abd98d38a24]) successfully deleted
I20260809 06:49:07.703660 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 17296
I20260809 06:49:07.703860 17180 catalog_manager.cc:4841] TS 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193:43779): tablet f817ea58d7174f2596604786f49a099b (table test-table [id=41880d8997764f1390186abd98d38a24]) successfully deleted
I20260809 06:49:07.734185 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 17499
Aug 09 06:49:07 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:49:07.750983 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.735080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.27.209.194:40675)
0809 06:49:07.735247 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.735252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.735271 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:49:07.735437 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.735441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.735514 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.735630 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:07.737292 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.738029 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.739920 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.740321 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.740416 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.745459 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:07.745473 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:07.745489 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:07.745521 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:07.748360 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.749138 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.749144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.749147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.749271 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.749648 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.749651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.749653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.749991 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:07.750202 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.750555 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.750718 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:49:07.751834 17678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:07.839936 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 17499
I20260809 06:49:07.862279 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 17690
Aug 09 06:49:07 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:49:07.878966 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:07.863183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.27.209.195:45389)
0809 06:49:07.863400 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:07.863407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:07.863430 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:49:07.863600 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:07.863606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:07.863746 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:07.863888 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:07.865169 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.866390 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.867609 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:07.868285 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:07.868442 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:07.873015 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:07.873027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:07.873044 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:07.873082 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:07.876023 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.876878 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.876885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.876887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.877036 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:07.877473 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:07.877476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:07.877478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:07.877824 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:07.878077 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:07.878455 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 06:49:07.878611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 06:49:07.879880 17878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:07.999307 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 17690
I20260809 06:49:08.023483 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 17156
I20260809 06:49:08.024794 17246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:08.133356 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 17156
I20260809 06:49:08.161134 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:32857
--webserver_interface=127.27.209.254
--webserver_port=33179
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:08.486796 17967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:08.487063 17967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:08.487200 17967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:08.494227 17967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:49:08.494325 17967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:08.494398 17967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:08.494475 17967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:49:08.494532 17967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:49:08.503912 17967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=33179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:08.505084 17967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:08.506358 17967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:08.531898 17984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:08.531927 17981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:08.531972 17979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:08.532886 17967 server_base.cc:1034] running on GCE node
I20260809 06:49:08.533746 17967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:08.535246 17967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:08.536444 17967 hybrid_clock.cc:648] HybridClock initialized: now 1786258148536374 us; error 52 us; skew 500 ppm
I20260809 06:49:08.536851 17967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:08 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:08.540328 17967 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:49:08.541909 17967 webserver.cc:460] Webserver started at http://127.27.209.254:33179/ using document root <none> and password file <none>
I20260809 06:49:08.542659 17967 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:08.542786 17967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:08.548554 17967 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 06:49:08.551391 17993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:08.552731 17967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:49:08.552892 17967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "3e6bb7897a2c40979e8450cdc191d23f"
format_stamp: "Formatted at 2026-08-09 06:49:05 on dist-test-slave-jpj1"
server_key: "e8aefa20cc29412a32db078ca4d86f81"
server_key_iv: "54e0a58da15eb1195f133271e652c8ef"
server_key_version: "encryptionkey@0"
I20260809 06:49:08.553911 17967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:08.566279 17967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:08.570456 17967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:08.570710 17967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:08.587882 17967 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:32857
I20260809 06:49:08.587905 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:32857 every 8 connection(s)
I20260809 06:49:08.589408 17967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:49:08.593915 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 17967
I20260809 06:49:08.594939 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:43779
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:32857
--webserver_port=38933
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:49:08.597563 18069 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:49:08.601958 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap starting.
I20260809 06:49:08.621997 18069 log.cc:824] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:08.634519 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:49:08.635046 18069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap complete.
I20260809 06:49:08.644196 18069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:08.645030 18069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6bb7897a2c40979e8450cdc191d23f, State: Initialized, Role: FOLLOWER
I20260809 06:49:08.645673 18069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:08.645912 18069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:49:08.646027 18069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:49:08.646189 18069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:49:08.649989 18069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:08.650502 18069 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6bb7897a2c40979e8450cdc191d23f; no voters:
I20260809 06:49:08.651144 18069 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:49:08.651300 18077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:49:08.652496 18077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 LEADER]: Becoming Leader. State: Replica: 3e6bb7897a2c40979e8450cdc191d23f, State: Running, Role: LEADER
I20260809 06:49:08.653102 18077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:08.653520 18069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:49:08.655789 18080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6bb7897a2c40979e8450cdc191d23f. Latest consensus state: current_term: 2 leader_uuid: "3e6bb7897a2c40979e8450cdc191d23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } } }
I20260809 06:49:08.656188 18080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:08.656106 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3e6bb7897a2c40979e8450cdc191d23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } } }
I20260809 06:49:08.656467 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:08.660096 18091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:49:08.668795 18091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:49:08.670433 18091 catalog_manager.cc:1259] Loaded cluster ID: 5aae535af97448fea313523d1b05f320
I20260809 06:49:08.670600 18091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:49:08.673956 18091 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 06:49:08.695066 18091 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 06:49:08.704589 18091 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:49:08.706444 18091 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Loaded TSK: 0
I20260809 06:49:08.708020 18091 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:49:08.945830 18074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:08.946094 18074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:08.946208 18074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:08.953274 18074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:08.953390 18074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:08.953544 18074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:49:08.963276 18074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:43779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=38933
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:08.964517 18074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:08.965826 18074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:08.980201 18116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:08.980852 18118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:08.980911 18115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:08.981345 18074 server_base.cc:1034] running on GCE node
I20260809 06:49:08.981925 18074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:08.982789 18074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:08.983990 18074 hybrid_clock.cc:648] HybridClock initialized: now 1786258148983927 us; error 55 us; skew 500 ppm
I20260809 06:49:08.984293 18074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:08 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:08.987426 18074 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:49:08.988902 18074 webserver.cc:460] Webserver started at http://127.27.209.193:38933/ using document root <none> and password file <none>
I20260809 06:49:08.989557 18074 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:08.989701 18074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:08.995038 18074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:08.997869 18129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:08.999303 18074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:08.999457 18074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "4e9b178312484c11b3f00ab19fee4807"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "e68a5e59e8df38999cbce7113781b745"
server_key_iv: "cdad22dcb4e0702a7b0b02b32a3da741"
server_key_version: "encryptionkey@0"
I20260809 06:49:09.000478 18074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:09.013406 18074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:09.017462 18074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:09.017716 18074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:09.018687 18074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:09.020143 18074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:09.020241 18074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.020350 18074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:09.020421 18074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.050485 18074 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:43779
I20260809 06:49:09.050552 18287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:43779 every 8 connection(s)
I20260809 06:49:09.052030 18074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:49:09.062633 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 18074
I20260809 06:49:09.063452 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:40675
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:32857
--webserver_port=36865
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:49:09 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258148, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:09.077097 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.055106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:34713)
0809 06:49:09.055936 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:09.055982 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:09.057697 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:09.059118 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:09.059128 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:09.059788 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:09.060315 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:09.060333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.061921 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.061924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:09.063851 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:09.063866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.065302 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.065309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:09.065591 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.066325 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:09.066352 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:09.068913 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:09.072416 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:09.072427 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:09.072443 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:09.072915 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:09.073391 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:09.073396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:09.073400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:09.073601 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:09.074506 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:09.074514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:09.074863 (+ 349us) client_negotiation.cc:769] Sending connection context
0809 06:49:09.075121 (+ 258us) client_negotiation.cc:240] Negotiation successful
0809 06:49:09.075744 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":321,"threads_started":1}
I20260809 06:49:09.077111 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.054873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:34713 (local address 127.27.209.254:32857)
0809 06:49:09.055420 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:09.055436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:09.056098 (+ 662us) server_negotiation.cc:407] Connection header received
0809 06:49:09.058252 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:09.058266 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:09.058615 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:09.058946 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:09.060532 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.061761 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.063992 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.065015 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.065216 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.069232 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:09.069241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:09.069258 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:09.069299 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:09.072277 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.073064 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.073071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.073078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.073272 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.073721 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.073725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.073727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.074358 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:09.074546 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:09.075390 (+ 844us) server_negotiation.cc:299] Negotiation successful
0809 06:49:09.075815 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":232,"threads_started":1}
I20260809 06:49:09.080068 18289 heartbeater.cc:347] Connected to a master server at 127.27.209.254:32857
I20260809 06:49:09.080420 18289 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:09.081375 18289 heartbeater.cc:511] Master 127.27.209.254:32857 requested a full tablet report, sending...
I20260809 06:49:09.083676 18024 ts_manager.cc:194] Registered new tserver with Master: 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193:43779)
I20260809 06:49:09.085192 18024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:34713
W20260809 06:49:09.392666 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:09.392926 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:09.393054 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:09.400166 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:09.400276 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:09.400424 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:49:09.412897 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:40675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=36865
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:09.414732 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:09.416533 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:09.430044 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:09.430011 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:09.429975 18302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:09.431121 18293 server_base.cc:1034] running on GCE node
I20260809 06:49:09.431735 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:09.432605 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:09.433796 18293 hybrid_clock.cc:648] HybridClock initialized: now 1786258149433741 us; error 49 us; skew 500 ppm
I20260809 06:49:09.434082 18293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:09 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:09.437111 18293 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:49:09.438508 18293 webserver.cc:460] Webserver started at http://127.27.209.194:36865/ using document root <none> and password file <none>
I20260809 06:49:09.439193 18293 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:09.439333 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:09.444494 18293 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:49:09.446986 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.448212 18293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 06:49:09.448369 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "578def645c8f4b87967295d852a2149f"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "7ca8fb2ea5c8aabf3c8ca92ee6f4c7ee"
server_key_iv: "0115a15bee69624b0559fe9a0f1e4de5"
server_key_version: "encryptionkey@0"
I20260809 06:49:09.449306 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:09.464332 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:09.468194 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:09.468415 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:09.469276 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:09.470526 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:09.470618 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.470695 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:09.470752 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.500492 18293 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:40675
I20260809 06:49:09.500561 18478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:40675 every 8 connection(s)
I20260809 06:49:09.501922 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:49:09.510344 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 18293
I20260809 06:49:09.510934 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:45389
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:32857
--webserver_port=39115
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:49:09 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:09.520533 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.504293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:58387 (local address 127.27.209.254:32857)
0809 06:49:09.504473 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:09.504477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:09.505659 (+ 1182us) server_negotiation.cc:407] Connection header received
0809 06:49:09.507415 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:09.507421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:09.507498 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:09.507635 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:09.509064 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.509783 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.511356 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.511815 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.511920 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.515263 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:09.515267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:09.515270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:09.515308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:09.517674 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.518346 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.518350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.518352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.518467 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.518867 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.518872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.518875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.519130 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:09.519252 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:09.519961 (+ 709us) server_negotiation.cc:299] Negotiation successful
0809 06:49:09.520194 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:49:09.521443 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.504834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:58387)
0809 06:49:09.505509 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:09.505552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:09.507160 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:09.507865 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:09.507873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:09.508345 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:09.508895 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:09.508913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.509942 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.509946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:09.511218 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:09.511231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.512034 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.512042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:09.512246 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.512849 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:09.512870 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:09.515061 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:09.517823 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:09.517832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:09.517846 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:09.518232 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:09.518577 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:09.518580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:09.518583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:09.518757 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:09.519262 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:09.519270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:09.519602 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 06:49:09.519850 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 06:49:09.520339 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":259,"threads_started":1}
I20260809 06:49:09.522886 18480 heartbeater.cc:347] Connected to a master server at 127.27.209.254:32857
I20260809 06:49:09.523159 18480 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:09.523907 18480 heartbeater.cc:511] Master 127.27.209.254:32857 requested a full tablet report, sending...
I20260809 06:49:09.525310 18024 ts_manager.cc:194] Registered new tserver with Master: 578def645c8f4b87967295d852a2149f (127.27.209.194:40675)
I20260809 06:49:09.526093 18024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:58387
W20260809 06:49:09.836500 18487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:09.836784 18487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:09.836920 18487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:09.843954 18487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:09.844063 18487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:09.844211 18487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:49:09.853523 18487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:45389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=39115
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:09.854784 18487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:09.856118 18487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:09.868585 18494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:09.868503 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:09.868902 18496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:09.869572 18487 server_base.cc:1034] running on GCE node
I20260809 06:49:09.870152 18487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:09.871014 18487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:09.872210 18487 hybrid_clock.cc:648] HybridClock initialized: now 1786258149872162 us; error 36 us; skew 500 ppm
I20260809 06:49:09.872509 18487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:09 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:09.875432 18487 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:49:09.876808 18487 webserver.cc:460] Webserver started at http://127.27.209.195:39115/ using document root <none> and password file <none>
I20260809 06:49:09.877453 18487 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:09.877585 18487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:09.882613 18487 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:49:09.885540 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.886868 18487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:49:09.887022 18487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "18ff6bb28e314a24b793eb4167f6cc16"
format_stamp: "Formatted at 2026-08-09 06:49:07 on dist-test-slave-jpj1"
server_key: "6408f9ffca3cbb79223af66ecae6e307"
server_key_iv: "0d0b4f34fee6a6bdae5f66a4ee2b1f21"
server_key_version: "encryptionkey@0"
I20260809 06:49:09.888037 18487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:09.906795 18487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:09.910501 18487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:09.910743 18487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:09.911718 18487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:09.912997 18487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:09.913087 18487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.913196 18487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:09.913270 18487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:09.942962 18487 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:45389
I20260809 06:49:09.943029 18665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:45389 every 8 connection(s)
I20260809 06:49:09.944389 18487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:49:09.946882 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 18487
Aug 09 06:49:09 dist-test-slave-jpj1 krb5kdc[17131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:09.965484 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.947746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.27.209.254:32857)
0809 06:49:09.951847 (+ 4101us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:09.951851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:09.951865 (+ 14us) server_negotiation.cc:407] Connection header received
0809 06:49:09.951907 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:09.951910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:09.951967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:09.952053 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:09.953775 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.954724 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.956092 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.956661 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.956803 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.959736 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:09.959743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:09.959748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:09.959786 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:09.962802 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.963597 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.963600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.963602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.963755 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.964113 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.964118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.964121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.964486 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:09.964700 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:09.964998 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 06:49:09.965162 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":3929,"thread_start_us":3955,"threads_started":1}
I20260809 06:49:09.966873 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.946805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:58043 (local address 127.27.209.254:32857)
0809 06:49:09.947015 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:09.947021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:09.948295 (+ 1274us) server_negotiation.cc:407] Connection header received
0809 06:49:09.951099 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:09.951105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:09.951199 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:09.951311 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:09.953168 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.954216 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.955994 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.956546 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.956647 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.960713 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:09.960718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:09.960720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:09.960760 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:09.963624 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.964381 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.964385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.964387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.964514 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:09.964976 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:09.964981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:09.964984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:09.965311 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:09.965443 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:09.966353 (+ 910us) server_negotiation.cc:299] Negotiation successful
0809 06:49:09.966547 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":68}
I20260809 06:49:09.967787 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.947302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:58043)
0809 06:49:09.948124 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:09.948183 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:09.950748 (+ 2565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:09.951910 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:09.951922 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:09.952450 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:09.953008 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:09.953024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.954389 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.954394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:09.955858 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:09.955871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.956796 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.956805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:09.957081 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.957752 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:09.957779 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:09.960313 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:09.963746 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:09.963754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:09.963770 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:09.964258 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:09.964622 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:09.964627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:09.964630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:09.964849 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:09.965511 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:09.965517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:09.965919 (+ 402us) client_negotiation.cc:769] Sending connection context
0809 06:49:09.966224 (+ 305us) client_negotiation.cc:240] Negotiation successful
0809 06:49:09.966603 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":335,"threads_started":1}
I20260809 06:49:09.968962 18666 heartbeater.cc:347] Connected to a master server at 127.27.209.254:32857
I20260809 06:49:09.969211 18666 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:09.969883 18666 heartbeater.cc:511] Master 127.27.209.254:32857 requested a full tablet report, sending...
I20260809 06:49:09.971146 18024 ts_manager.cc:194] Registered new tserver with Master: 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195:45389)
I20260809 06:49:09.972007 18024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:58043
I20260809 06:49:09.980073 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:49:09.986697 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:09.980998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.27.209.254:32857)
0809 06:49:09.981153 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:09.981159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:09.981560 (+ 401us) server_negotiation.cc:407] Connection header received
0809 06:49:09.981761 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:09.981764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:09.981893 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:09.982015 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:49:09.982437 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.983170 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.984745 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:09.985143 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:09.985250 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:09.985737 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:49:09.986166 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:49:09.986292 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:09.986399 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 06:49:09.986475 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 06:49:09.987622 18024 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:49:09.994578 18024 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:49:10.016321 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.005831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:43779 (local address 127.0.0.1:42690)
0809 06:49:10.006916 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:10.006972 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:10.007149 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:10.009556 (+ 2407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:10.009563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:10.009614 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:10.009968 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.009981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.011611 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.011617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.013675 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.013691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.014968 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.014975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.015157 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.015194 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:49:10.015312 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 06:49:10.015496 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":37,"thread_start_us":1461,"threads_started":1}
I20260809 06:49:10.017043 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.006077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.27.209.193:43779)
0809 06:49:10.008859 (+ 2782us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.008865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.008887 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:49:10.008962 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.008967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.009147 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.009557 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:10.010156 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.011475 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.013827 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.014749 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.014908 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.015092 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.015374 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.016702 (+ 1328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2645,"thread_start_us":2534,"threads_started":1}
I20260809 06:49:10.018013 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.006742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.27.209.194:40675)
0809 06:49:10.007893 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.007903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.008477 (+ 574us) server_negotiation.cc:407] Connection header received
0809 06:49:10.009524 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.009532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.010093 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.010271 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:10.010859 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.012365 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.015043 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.016504 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.016684 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.016852 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.017516 (+ 664us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.017683 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":875,"threads_started":1}
I20260809 06:49:10.018049 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.006270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40675 (local address 127.0.0.1:38920)
0809 06:49:10.008363 (+ 2093us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:10.008381 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:10.009352 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:10.010267 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:10.010271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:10.010302 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:10.010692 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.010703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.012532 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.012537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.014851 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.014867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.017063 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.017069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.017307 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.017356 (+ 49us) client_negotiation.cc:769] Sending connection context
0809 06:49:10.017448 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 06:49:10.017599 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1968,"mutex_wait_us":24,"thread_start_us":1794,"threads_started":1}
I20260809 06:49:10.018071 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.006498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:45389 (local address 127.0.0.1:52470)
0809 06:49:10.009165 (+ 2667us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:10.009181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:10.009309 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:10.010439 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:10.010443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:10.010471 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:10.010882 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.010894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.012792 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.012797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.014918 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.014929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.017043 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.017050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.017325 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.017371 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 06:49:10.017448 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 06:49:10.017599 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2542,"mutex_wait_us":102,"thread_start_us":2388,"threads_started":1}
I20260809 06:49:10.018735 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.007131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.27.209.195:45389)
0809 06:49:10.008715 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.008724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.009624 (+ 900us) server_negotiation.cc:407] Connection header received
0809 06:49:10.009742 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.009749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.009983 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.010152 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:10.011077 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.012629 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.015077 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.016504 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.016684 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.016852 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.018267 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.018433 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1413,"thread_start_us":1273,"threads_started":1}
I20260809 06:49:10.021111 18197 tablet_service.cc:1467] Processing CreateTablet for tablet 54e80721a18c47c48076d39598e38db7 (DEFAULT_TABLE table=test-table [id=bfea9bdba6b54abd979e9cca1abcd074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:10.022267 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e80721a18c47c48076d39598e38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:10.021800 18569 tablet_service.cc:1467] Processing CreateTablet for tablet 54e80721a18c47c48076d39598e38db7 (DEFAULT_TABLE table=test-table [id=bfea9bdba6b54abd979e9cca1abcd074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:10.022992 18569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e80721a18c47c48076d39598e38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:10.027207 18384 tablet_service.cc:1467] Processing CreateTablet for tablet 54e80721a18c47c48076d39598e38db7 (DEFAULT_TABLE table=test-table [id=bfea9bdba6b54abd979e9cca1abcd074]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:10.028594 18384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e80721a18c47c48076d39598e38db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:10.035279 18687 tablet_bootstrap.cc:492] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Bootstrap starting.
I20260809 06:49:10.038733 18687 tablet_bootstrap.cc:654] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:10.040123 18687 log.cc:824] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:10.040249 18688 tablet_bootstrap.cc:492] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Bootstrap starting.
I20260809 06:49:10.043397 18691 tablet_bootstrap.cc:492] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Bootstrap starting.
I20260809 06:49:10.043752 18687 tablet_bootstrap.cc:492] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: No bootstrap required, opened a new log
I20260809 06:49:10.043751 18688 tablet_bootstrap.cc:654] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:10.044106 18687 ts_tablet_manager.cc:1397] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 06:49:10.045004 18688 log.cc:824] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:10.046716 18691 tablet_bootstrap.cc:654] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:10.048017 18691 log.cc:824] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:10.048992 18688 tablet_bootstrap.cc:492] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: No bootstrap required, opened a new log
I20260809 06:49:10.049232 18688 ts_tablet_manager.cc:1397] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260809 06:49:10.050954 18691 tablet_bootstrap.cc:492] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: No bootstrap required, opened a new log
I20260809 06:49:10.051245 18691 ts_tablet_manager.cc:1397] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 06:49:10.053299 18687 raft_consensus.cc:348] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.053817 18687 raft_consensus.cc:374] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:10.053936 18687 raft_consensus.cc:729] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ff6bb28e314a24b793eb4167f6cc16, State: Initialized, Role: FOLLOWER
I20260809 06:49:10.054533 18687 consensus_queue.cc:260] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.061053 18691 raft_consensus.cc:348] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.061033 18688 raft_consensus.cc:348] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.061457 18688 raft_consensus.cc:374] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:10.061507 18691 raft_consensus.cc:374] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:10.061594 18688 raft_consensus.cc:729] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b178312484c11b3f00ab19fee4807, State: Initialized, Role: FOLLOWER
I20260809 06:49:10.061654 18691 raft_consensus.cc:729] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578def645c8f4b87967295d852a2149f, State: Initialized, Role: FOLLOWER
I20260809 06:49:10.061684 18666 heartbeater.cc:503] Master 127.27.209.254:32857 was elected leader, sending a full tablet report...
I20260809 06:49:10.062260 18688 consensus_queue.cc:260] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.062271 18691 consensus_queue.cc:260] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.062858 18687 ts_tablet_manager.cc:1428] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Time spent starting tablet: real 0.019s user 0.007s sys 0.011s
I20260809 06:49:10.064090 18480 heartbeater.cc:503] Master 127.27.209.254:32857 was elected leader, sending a full tablet report...
I20260809 06:49:10.064209 18289 heartbeater.cc:503] Master 127.27.209.254:32857 was elected leader, sending a full tablet report...
I20260809 06:49:10.064426 18691 ts_tablet_manager.cc:1428] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260809 06:49:10.064810 18688 ts_tablet_manager.cc:1428] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260809 06:49:10.075512 18697 raft_consensus.cc:482] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:49:10.075799 18697 raft_consensus.cc:504] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.077308 18697 leader_election.cc:288] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 578def645c8f4b87967295d852a2149f (127.27.209.194:40675), 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195:45389)
I20260809 06:49:10.083498 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.078171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:38187 (local address 127.27.209.194:40675)
0809 06:49:10.078368 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.078373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.078552 (+ 179us) server_negotiation.cc:407] Connection header received
0809 06:49:10.078693 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.078697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.078750 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.078843 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:10.079441 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.080155 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.081889 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.082648 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.082807 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.082882 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.083150 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.083266 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:49:10.083608 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.077729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:37151 (local address 127.27.209.195:45389)
0809 06:49:10.077922 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.077929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.078308 (+ 379us) server_negotiation.cc:407] Connection header received
0809 06:49:10.078570 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.078574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.078632 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.078706 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:10.079297 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.080046 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.081911 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.082710 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.082858 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.082928 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.083227 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.083347 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:49:10.083750 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.077565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40675 (local address 127.27.209.193:38187)
0809 06:49:10.078455 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:10.078470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:10.078586 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:10.078858 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:10.078862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:10.078899 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:10.079302 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.079311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.080282 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.080297 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.081754 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.081766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.082843 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.082850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.083008 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.083035 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 06:49:10.083114 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 06:49:10.083238 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":22,"thread_start_us":615,"threads_started":1}
I20260809 06:49:10.083789 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.077546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:45389 (local address 127.27.209.193:37151)
0809 06:49:10.078193 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:10.078211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:10.078424 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:10.078742 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:10.078747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:10.078778 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:10.079151 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.079165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.080178 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.080182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.081761 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:10.081774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.082923 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.082929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:10.083078 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.083100 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:49:10.083177 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 06:49:10.083297 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":87,"thread_start_us":306,"threads_started":1}
I20260809 06:49:10.084826 18415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e80721a18c47c48076d39598e38db7" candidate_uuid: "4e9b178312484c11b3f00ab19fee4807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578def645c8f4b87967295d852a2149f" is_pre_election: true
I20260809 06:49:10.084830 18597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e80721a18c47c48076d39598e38db7" candidate_uuid: "4e9b178312484c11b3f00ab19fee4807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ff6bb28e314a24b793eb4167f6cc16" is_pre_election: true
I20260809 06:49:10.085263 18597 raft_consensus.cc:2454] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9b178312484c11b3f00ab19fee4807 in term 0.
I20260809 06:49:10.085263 18415 raft_consensus.cc:2454] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9b178312484c11b3f00ab19fee4807 in term 0.
I20260809 06:49:10.085881 18133 leader_election.cc:302] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b178312484c11b3f00ab19fee4807, 578def645c8f4b87967295d852a2149f; no voters:
I20260809 06:49:10.086248 18697 raft_consensus.cc:2781] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:49:10.086406 18697 raft_consensus.cc:482] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:49:10.086510 18697 raft_consensus.cc:3037] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:10.088770 18697 raft_consensus.cc:504] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.089398 18697 leader_election.cc:288] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [CANDIDATE]: Term 1 election: Requested vote from peers 578def645c8f4b87967295d852a2149f (127.27.209.194:40675), 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195:45389)
I20260809 06:49:10.089771 18415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e80721a18c47c48076d39598e38db7" candidate_uuid: "4e9b178312484c11b3f00ab19fee4807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578def645c8f4b87967295d852a2149f"
I20260809 06:49:10.089905 18597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e80721a18c47c48076d39598e38db7" candidate_uuid: "4e9b178312484c11b3f00ab19fee4807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ff6bb28e314a24b793eb4167f6cc16"
I20260809 06:49:10.089960 18415 raft_consensus.cc:3037] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:10.090101 18597 raft_consensus.cc:3037] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:10.092343 18415 raft_consensus.cc:2454] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9b178312484c11b3f00ab19fee4807 in term 1.
I20260809 06:49:10.092456 18597 raft_consensus.cc:2454] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9b178312484c11b3f00ab19fee4807 in term 1.
I20260809 06:49:10.092715 18133 leader_election.cc:302] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9b178312484c11b3f00ab19fee4807, 578def645c8f4b87967295d852a2149f; no voters:
I20260809 06:49:10.092986 18697 raft_consensus.cc:2781] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:10.093433 18697 raft_consensus.cc:686] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9b178312484c11b3f00ab19fee4807, State: Running, Role: LEADER
I20260809 06:49:10.093873 18697 consensus_queue.cc:237] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } }
I20260809 06:49:10.098616 18024 catalog_manager.cc:5495] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "4e9b178312484c11b3f00ab19fee4807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9b178312484c11b3f00ab19fee4807" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 43779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 } health_report { overall_health: UNKNOWN } } }
I20260809 06:49:10.118862 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.113600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.27.209.193:43779)
0809 06:49:10.113802 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.113808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.113871 (+ 63us) server_negotiation.cc:407] Connection header received
0809 06:49:10.114094 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.114098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.114182 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.114317 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:49:10.114672 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.115411 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.116827 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.117220 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.117339 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.117809 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:49:10.118296 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:49:10.118428 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.118518 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.118597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:49:10.127249 18415 raft_consensus.cc:1261] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Refusing update from remote peer 4e9b178312484c11b3f00ab19fee4807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:10.127249 18597 raft_consensus.cc:1261] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Refusing update from remote peer 4e9b178312484c11b3f00ab19fee4807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:10.128376 18700 consensus_queue.cc:1035] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "578def645c8f4b87967295d852a2149f" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:10.128702 18697 consensus_queue.cc:1035] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ff6bb28e314a24b793eb4167f6cc16" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:10.154747 18022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:10.155021 18022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:10.158625 18022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Sending DeleteTablet for 3 replicas of tablet 54e80721a18c47c48076d39598e38db7
I20260809 06:49:10.159802 18197 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e80721a18c47c48076d39598e38db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:10 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42690
I20260809 06:49:10.159884 18384 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e80721a18c47c48076d39598e38db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:10 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:38920
I20260809 06:49:10.160432 18569 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e80721a18c47c48076d39598e38db7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:10 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:52470
I20260809 06:49:10.160755 18716 tablet_replica.cc:330] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: stopping tablet replica
I20260809 06:49:10.161170 18716 raft_consensus.cc:2229] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:10.161320 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 18074
I20260809 06:49:10.161662 18716 raft_consensus.cc:2258] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:10.163182 18715 tablet_replica.cc:330] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: stopping tablet replica
I20260809 06:49:10.163672 18715 raft_consensus.cc:2229] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:49:10.164366 18715 raft_consensus.cc:2258] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:10.168013 18718 tablet_replica.cc:330] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: stopping tablet replica
I20260809 06:49:10.168000 18715 ts_tablet_manager.cc:1905] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:10.168406 18718 raft_consensus.cc:2229] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:10.168881 18718 raft_consensus.cc:2258] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:10.169143 18716 ts_tablet_manager.cc:1905] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:10.175814 18715 ts_tablet_manager.cc:1918] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:10.176007 18715 log.cc:1196] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/54e80721a18c47c48076d39598e38db7
I20260809 06:49:10.176594 18715 ts_tablet_manager.cc:1939] T 54e80721a18c47c48076d39598e38db7 P 4e9b178312484c11b3f00ab19fee4807: Deleting consensus metadata
I20260809 06:49:10.178458 17995 catalog_manager.cc:4841] TS 4e9b178312484c11b3f00ab19fee4807 (127.27.209.193:43779): tablet 54e80721a18c47c48076d39598e38db7 (table test-table [id=bfea9bdba6b54abd979e9cca1abcd074]) successfully deleted
I20260809 06:49:10.179147 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.162136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.27.209.193:43779)
0809 06:49:10.162309 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.162316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.162450 (+ 134us) server_negotiation.cc:407] Connection header received
0809 06:49:10.162635 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.162640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.162738 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.162844 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:10.164627 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.165834 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.167869 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.168484 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.168613 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.171219 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:10.171231 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:10.171247 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:10.171294 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:10.175973 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:10.176819 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:10.176837 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:10.176840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:10.177012 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:10.177397 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:10.177401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:10.177404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:10.177872 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:10.178205 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.178484 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.178777 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:49:10.179900 18281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:10.181216 18718 ts_tablet_manager.cc:1905] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:10.183822 18716 ts_tablet_manager.cc:1918] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:10.184104 18716 log.cc:1196] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/54e80721a18c47c48076d39598e38db7
I20260809 06:49:10.184693 18716 ts_tablet_manager.cc:1939] T 54e80721a18c47c48076d39598e38db7 P 578def645c8f4b87967295d852a2149f: Deleting consensus metadata
I20260809 06:49:10.186638 17998 catalog_manager.cc:4841] TS 578def645c8f4b87967295d852a2149f (127.27.209.194:40675): tablet 54e80721a18c47c48076d39598e38db7 (table test-table [id=bfea9bdba6b54abd979e9cca1abcd074]) successfully deleted
I20260809 06:49:10.187896 18718 ts_tablet_manager.cc:1918] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:10.188042 18718 log.cc:1196] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/54e80721a18c47c48076d39598e38db7
I20260809 06:49:10.188499 18718 ts_tablet_manager.cc:1939] T 54e80721a18c47c48076d39598e38db7 P 18ff6bb28e314a24b793eb4167f6cc16: Deleting consensus metadata
I20260809 06:49:10.190028 17997 catalog_manager.cc:4841] TS 18ff6bb28e314a24b793eb4167f6cc16 (127.27.209.195:45389): tablet 54e80721a18c47c48076d39598e38db7 (table test-table [id=bfea9bdba6b54abd979e9cca1abcd074]) successfully deleted
I20260809 06:49:10.275432 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 18074
I20260809 06:49:10.297798 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 18293
I20260809 06:49:10.310204 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.298541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.27.209.194:40675)
0809 06:49:10.298706 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.298710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.298728 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:49:10.298853 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.298857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.298933 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.299015 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:10.300363 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.301095 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.302294 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.302724 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.302854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.305352 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:10.305366 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:10.305382 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:10.305413 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:10.307992 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:10.308568 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:10.308574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:10.308576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:10.308690 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:10.308974 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:10.308976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:10.308978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:10.309350 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:10.309564 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.309787 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.309947 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:49:10.310915 18474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:10.397320 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 18293
I20260809 06:49:10.420794 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 18487
I20260809 06:49:10.434059 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:10.421576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.27.209.195:45389)
0809 06:49:10.421793 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:10.421799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:10.421821 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:49:10.421892 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:10.421897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:10.421977 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:10.422064 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:10.423435 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.424234 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.425511 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:10.425924 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:10.426033 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:10.428715 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:10.428730 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:10.428749 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:10.428781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:10.431277 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:10.432013 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:10.432023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:10.432027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:10.432160 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:10.432515 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:10.432520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:10.432524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:10.432894 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:10.433169 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:10.433507 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 06:49:10.433741 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:49:10.434712 18659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:10.520259 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 18487
I20260809 06:49:10.541893 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 17967
I20260809 06:49:10.543017 18064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:10.654461 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 17967
I20260809 06:49:10.681193 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:32857
--webserver_interface=127.27.209.254
--webserver_port=33179
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:11.009696 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:11.010005 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:11.010147 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:11.017302 18725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:49:11.017428 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:11.017511 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:11.017596 18725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:49:11.017668 18725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:49:11.027613 18725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=33179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:11.028941 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:11.030265 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:11.041414 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:11.041378 18740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:11.041456 18735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:11.042580 18725 server_base.cc:1034] running on GCE node
I20260809 06:49:11.043340 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:11.044742 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:11.045939 18725 hybrid_clock.cc:648] HybridClock initialized: now 1786258151045889 us; error 38 us; skew 500 ppm
I20260809 06:49:11.046288 18725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:11 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258151, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:11.049280 18725 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:49:11.050673 18725 webserver.cc:460] Webserver started at http://127.27.209.254:33179/ using document root <none> and password file <none>
I20260809 06:49:11.051349 18725 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:11.051463 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:11.056700 18725 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:49:11.059402 18749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:11.060671 18725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:49:11.060827 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "3e6bb7897a2c40979e8450cdc191d23f"
format_stamp: "Formatted at 2026-08-09 06:49:05 on dist-test-slave-jpj1"
server_key: "e8aefa20cc29412a32db078ca4d86f81"
server_key_iv: "54e0a58da15eb1195f133271e652c8ef"
server_key_version: "encryptionkey@0"
I20260809 06:49:11.061753 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:11.078415 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:11.082482 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:11.082737 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:11.100746 18725 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:32857
I20260809 06:49:11.100766 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:32857 every 8 connection(s)
I20260809 06:49:11.102273 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:49:11.104952 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 18725
I20260809 06:49:11.105976 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:43779
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:32857
--webserver_port=38933
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:49:11.110940 18832 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:49:11.115357 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap starting.
I20260809 06:49:11.135190 18832 log.cc:824] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:11.151221 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:49:11.151757 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f: Bootstrap complete.
I20260809 06:49:11.160451 18832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:11.161270 18832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6bb7897a2c40979e8450cdc191d23f, State: Initialized, Role: FOLLOWER
I20260809 06:49:11.161849 18832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:11.162051 18832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:49:11.162160 18832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:49:11.162326 18832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 2 FOLLOWER]: Advancing to term 3
I20260809 06:49:11.166056 18832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:11.166533 18832 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6bb7897a2c40979e8450cdc191d23f; no voters:
I20260809 06:49:11.167120 18832 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 06:49:11.167260 18836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 06:49:11.169179 18832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:49:11.171815 18836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [term 3 LEADER]: Becoming Leader. State: Replica: 3e6bb7897a2c40979e8450cdc191d23f, State: Running, Role: LEADER
I20260809 06:49:11.172490 18836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } }
I20260809 06:49:11.175205 18837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3e6bb7897a2c40979e8450cdc191d23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } } }
I20260809 06:49:11.175275 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6bb7897a2c40979e8450cdc191d23f. Latest consensus state: current_term: 3 leader_uuid: "3e6bb7897a2c40979e8450cdc191d23f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6bb7897a2c40979e8450cdc191d23f" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 32857 } } }
I20260809 06:49:11.175681 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:11.175688 18837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6bb7897a2c40979e8450cdc191d23f [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:11.184702 18851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:49:11.193501 18851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:49:11.195132 18851 catalog_manager.cc:1259] Loaded cluster ID: 5aae535af97448fea313523d1b05f320
I20260809 06:49:11.195325 18851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 06:49:11.197571 18851 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb649bef9ad google::LogMessage::Fail() at ??:0
@ 0x7fb649bf288a google::LogMessage::SendToLog() at ??:0
@ 0x7fb649bef4de google::LogMessage::Flush() at ??:0
@ 0x7fb649bf0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb65f622512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb65f61f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb65f6a7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb65f6a7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb64c83b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb64c83e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb64c83e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb64c819eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb64faf36db start_thread at ??:0
@ 0x7fb648a8961f clone at ??:0
W20260809 06:49:11.478013 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:11.478292 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:11.478425 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:11.485553 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:11.485688 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:11.485832 18834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:49:11.495388 18834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:43779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=38933
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:11.496647 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:11.497951 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:11.510675 18884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:11.510818 18881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:11.510885 18882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:11.511740 18834 server_base.cc:1034] running on GCE node
I20260809 06:49:11.512317 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:11.513195 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:11.514415 18834 hybrid_clock.cc:648] HybridClock initialized: now 1786258151514359 us; error 51 us; skew 500 ppm
I20260809 06:49:11.514714 18834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:11 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258151, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:11.517709 18834 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:49:11.519125 18834 webserver.cc:460] Webserver started at http://127.27.209.193:38933/ using document root <none> and password file <none>
I20260809 06:49:11.519855 18834 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:11.519977 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:11.525031 18834 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:49:11.527613 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:11.528903 18834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:49:11.529068 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "4e9b178312484c11b3f00ab19fee4807"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "e68a5e59e8df38999cbce7113781b745"
server_key_iv: "cdad22dcb4e0702a7b0b02b32a3da741"
server_key_version: "encryptionkey@0"
I20260809 06:49:11.530056 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:11.544190 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:11.547847 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:11.548107 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:11.549010 18834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:11.550302 18834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:11.550396 18834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:11.550503 18834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:11.550575 18834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:11.581866 18834 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:43779
I20260809 06:49:11.581938 19067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:43779 every 8 connection(s)
I20260809 06:49:11.583287 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:49:11.583954 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 18834
I20260809 06:49:11.584733 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:40675
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:32857
--webserver_port=36865
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:49:11.588378 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:11.586329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:56185)
0809 06:49:11.587177 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:11.587449 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":318,"threads_started":1}
W20260809 06:49:11.589148 19068 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:32857 (0 consecutive failures): Network error: Failed to ping master at 127.27.209.254:32857: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
I20260809 06:49:11.589998 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:11.589678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:40569)
0809 06:49:11.589825 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:11.589894 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 06:49:11.590847 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:11.590528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:38215)
0809 06:49:11.590662 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:11.590740 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:49:11.591115 19068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:49:11.912935 19072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:11.913210 19072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:11.913348 19072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:11.920419 19072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:11.920521 19072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:11.920671 19072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:49:11.930001 19072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:40675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=36865
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:11.931241 19072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:11.932564 19072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:11.945410 19078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:11.945413 19079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:11.945387 19081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:11.946081 19072 server_base.cc:1034] running on GCE node
I20260809 06:49:11.946689 19072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:11.947568 19072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:11.948783 19072 hybrid_clock.cc:648] HybridClock initialized: now 1786258151948726 us; error 50 us; skew 500 ppm
I20260809 06:49:11.949084 19072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:11 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258151, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:11.954843 19072 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:49:11.956290 19072 webserver.cc:460] Webserver started at http://127.27.209.194:36865/ using document root <none> and password file <none>
I20260809 06:49:11.956939 19072 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:11.957055 19072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:11.962201 19072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 06:49:11.964908 19088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:11.966193 19072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:49:11.966352 19072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "578def645c8f4b87967295d852a2149f"
format_stamp: "Formatted at 2026-08-09 06:49:06 on dist-test-slave-jpj1"
server_key: "7ca8fb2ea5c8aabf3c8ca92ee6f4c7ee"
server_key_iv: "0115a15bee69624b0559fe9a0f1e4de5"
server_key_version: "encryptionkey@0"
I20260809 06:49:11.967303 19072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:11.982339 19072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:11.985877 19072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:11.986119 19072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:11.987010 19072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:11.988346 19072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:11.988449 19072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:11.988554 19072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:11.988622 19072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:12.018496 19072 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:40675
I20260809 06:49:12.018563 19245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:40675 every 8 connection(s)
I20260809 06:49:12.019972 19072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:49:12.024617 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.022818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:37449)
0809 06:49:12.023557 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.023794 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
W20260809 06:49:12.025193 19246 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:32857 (0 consecutive failures): Network error: Failed to ping master at 127.27.209.254:32857: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
I20260809 06:49:12.025911 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.025625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:56069)
0809 06:49:12.025745 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.025810 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 06:49:12.026605 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.026341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:54891)
0809 06:49:12.026443 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.026510 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 06:49:12.026804 19246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:49:12.029409 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 19072
I20260809 06:49:12.030045 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:45389
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:32857
--webserver_port=39115
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:37973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:12.361304 19249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:12.361584 19249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:12.361677 19249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:12.368719 19249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:12.368830 19249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:12.368950 19249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:49:12.378409 19249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:37973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:45389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=39115
--tserver_master_addrs=127.27.209.254:32857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:12.379639 19249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:12.380968 19249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:12.393841 19259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:12.393863 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:12.393900 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:12.394672 19249 server_base.cc:1034] running on GCE node
I20260809 06:49:12.395275 19249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:12.396209 19249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:12.397406 19249 hybrid_clock.cc:648] HybridClock initialized: now 1786258152397364 us; error 32 us; skew 500 ppm
I20260809 06:49:12.397689 19249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:12 dist-test-slave-jpj1 krb5kdc[17131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258152, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:12.400759 19249 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:49:12.402341 19249 webserver.cc:460] Webserver started at http://127.27.209.195:39115/ using document root <none> and password file <none>
I20260809 06:49:12.403007 19249 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:12.403119 19249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:12.408430 19249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:49:12.411104 19269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:12.412478 19249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:49:12.412636 19249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "18ff6bb28e314a24b793eb4167f6cc16"
format_stamp: "Formatted at 2026-08-09 06:49:07 on dist-test-slave-jpj1"
server_key: "6408f9ffca3cbb79223af66ecae6e307"
server_key_iv: "0d0b4f34fee6a6bdae5f66a4ee2b1f21"
server_key_version: "encryptionkey@0"
I20260809 06:49:12.413592 19249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:12.445816 19249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:12.450035 19249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:12.450304 19249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:12.451346 19249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:12.452759 19249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:12.452860 19249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:12.452975 19249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:12.453052 19249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:12.484112 19249 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:45389
I20260809 06:49:12.484169 19451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:45389 every 8 connection(s)
I20260809 06:49:12.485577 19249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:49:12.486912 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 19249
I20260809 06:49:12.490584 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.488813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:57207)
0809 06:49:12.489500 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.489768 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":257,"threads_started":1}
W20260809 06:49:12.491174 19453 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:32857 (0 consecutive failures): Network error: Failed to ping master at 127.27.209.254:32857: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
I20260809 06:49:12.491946 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.491632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:36519)
0809 06:49:12.491771 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.491839 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 06:49:12.492734 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.492386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:60389)
0809 06:49:12.492539 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.492620 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 06:49:12.492965 19453 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:49:12.592419 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:12.591609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:55377)
0809 06:49:12.592065 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:12.592199 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":222,"threads_started":1}
I20260809 06:49:13.028154 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:13.027308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:57295)
0809 06:49:13.027803 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:13.027928 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":266,"threads_started":1}
I20260809 06:49:13.494184 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:13.493412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:46025)
0809 06:49:13.493854 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:13.493965 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":210,"threads_started":1}
I20260809 06:49:13.593986 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:13.593245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:54013)
0809 06:49:13.593729 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:13.593818 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
I20260809 06:49:14.029842 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:14.029054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:52701)
0809 06:49:14.029560 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:14.029660 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
I20260809 06:49:14.495672 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:14.494989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:42497)
0809 06:49:14.495401 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:14.495481 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":197,"threads_started":1}
I20260809 06:49:14.595502 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:14.594791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:56677)
0809 06:49:14.595242 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:14.595326 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260809 06:49:15.031448 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:15.030687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:45183)
0809 06:49:15.031184 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:15.031275 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":284,"threads_started":1}
I20260809 06:49:15.497195 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:15.496504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:57491)
0809 06:49:15.496940 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:15.497023 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260809 06:49:15.601202 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:15.598443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:40307)
0809 06:49:15.600035 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:15.600607 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":757,"threads_started":1}
I20260809 06:49:16.034790 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:16.032887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:40403)
0809 06:49:16.034051 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:16.034268 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":570,"threads_started":1}
I20260809 06:49:16.498793 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:16.498015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:55085)
0809 06:49:16.498538 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:16.498621 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":247,"threads_started":1}
I20260809 06:49:16.603793 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:16.602991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:42975)
0809 06:49:16.603494 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:16.603607 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":226,"threads_started":1}
I20260809 06:49:17.036670 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:17.035954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:47359)
0809 06:49:17.036425 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:17.036500 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260809 06:49:17.500429 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:17.499648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:42369)
0809 06:49:17.500155 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:17.500242 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":247,"threads_started":1}
I20260809 06:49:17.605263 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:17.604548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:48949)
0809 06:49:17.605004 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:17.605091 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260809 06:49:18.038229 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:18.037480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:46073)
0809 06:49:18.037966 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:18.038057 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
I20260809 06:49:18.501996 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:18.501244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:37639)
0809 06:49:18.501676 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:18.501786 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260809 06:49:18.606881 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:18.606105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:38267)
0809 06:49:18.606571 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:18.606651 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":211,"threads_started":1}
I20260809 06:49:19.040112 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:19.039213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:37679)
0809 06:49:19.039789 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:19.039875 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
I20260809 06:49:19.503746 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:19.502874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:44303)
0809 06:49:19.503354 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:19.503448 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":214,"threads_started":1}
I20260809 06:49:19.608333 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:19.607704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:36009)
0809 06:49:19.608100 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:19.608177 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":197,"threads_started":1}
I20260809 06:49:20.041980 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:20.041129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:43203)
0809 06:49:20.041646 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:20.041775 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
I20260809 06:49:20.505182 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:20.504520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:59401)
0809 06:49:20.504950 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:20.505027 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":223,"threads_started":1}
I20260809 06:49:20.609911 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:20.609162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:55089)
0809 06:49:20.609652 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:20.609735 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260809 06:49:21.043704 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:21.042895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:40421)
0809 06:49:21.043394 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:21.043480 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":233,"threads_started":1}
I20260809 06:49:21.506739 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:21.505986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:41231)
0809 06:49:21.506463 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:21.506571 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":232,"threads_started":1}
I20260809 06:49:21.611440 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:21.610739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:38135)
0809 06:49:21.611183 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:21.611264 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260809 06:49:22.045378 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:22.044623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:48665)
0809 06:49:22.045108 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:22.045198 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":267,"threads_started":1}
I20260809 06:49:22.508360 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:22.507680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:33307)
0809 06:49:22.508109 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:22.508188 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20260809 06:49:22.613183 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:22.612393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:58335)
0809 06:49:22.612885 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:22.612997 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":277,"threads_started":1}
I20260809 06:49:23.047160 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:23.046318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:33169)
0809 06:49:23.046880 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:23.046976 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":345,"threads_started":1}
I20260809 06:49:23.510011 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:23.509184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:42139)
0809 06:49:23.509675 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:23.509788 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
I20260809 06:49:23.614711 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:23.614029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:45643)
0809 06:49:23.614441 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:23.614522 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":216,"threads_started":1}
I20260809 06:49:24.048820 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:24.048114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:58369)
0809 06:49:24.048562 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:24.048653 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260809 06:49:24.511623 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:24.510899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:53925)
0809 06:49:24.511330 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:24.511423 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260809 06:49:24.616312 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:24.615516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:59551)
0809 06:49:24.616007 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:24.616100 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":232,"threads_started":1}
I20260809 06:49:25.050575 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:25.049810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:60837)
0809 06:49:25.050294 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:25.050387 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":226,"threads_started":1}
I20260809 06:49:25.513334 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:25.512475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:37799)
0809 06:49:25.512989 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:25.513080 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":264,"threads_started":1}
I20260809 06:49:25.617938 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:25.617097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:50711)
0809 06:49:25.617657 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:25.617750 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260809 06:49:26.052287 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:26.051483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:54275)
0809 06:49:26.051991 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:26.052089 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260809 06:49:26.514838 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:26.514131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:58523)
0809 06:49:26.514556 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:26.514645 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260809 06:49:26.619772 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:26.619014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.193:36563)
0809 06:49:26.619452 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:26.619582 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260809 06:49:27.054124 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:27.053377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.194:48883)
0809 06:49:27.053835 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:27.053935 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260809 06:49:27.489171 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 18834
I20260809 06:49:27.512377 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:27.490376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.27.209.193:43779)
0809 06:49:27.490999 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:27.491017 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:27.491295 (+ 278us) server_negotiation.cc:407] Connection header received
0809 06:49:27.493625 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:27.493635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:27.494079 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:27.494376 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:27.496071 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:27.497409 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:27.498627 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:27.499287 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:27.499422 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:27.506209 (+ 6787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:27.506217 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:27.506233 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:27.506272 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:27.509221 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:27.509942 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:27.509949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:27.509956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:27.510129 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:27.510498 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:27.510502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:27.510503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:27.511033 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:27.511262 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:27.511705 (+ 443us) server_negotiation.cc:299] Negotiation successful
0809 06:49:27.512098 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":284,"threads_started":1}
I20260809 06:49:27.513819 19061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:27.518496 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:27.516060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:32857 (local address 127.27.209.195:53135)
0809 06:49:27.518219 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:27.518324 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:32857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1925,"threads_started":1}
I20260809 06:49:27.608690 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 18834
I20260809 06:49:27.635849 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 19072
I20260809 06:49:27.656352 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:27.636854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.27.209.194:40675)
0809 06:49:27.637402 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:27.637421 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:27.637465 (+ 44us) server_negotiation.cc:407] Connection header received
0809 06:49:27.639925 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:27.639939 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:27.640493 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:27.640885 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:27.642691 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:27.644066 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:27.645940 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:27.646650 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:27.646785 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:27.650284 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:27.650292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:27.650307 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:27.650343 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:27.653378 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:27.654064 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:27.654071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:27.654077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:27.654244 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:27.654557 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:27.654560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:27.654562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:27.655044 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:27.655286 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:27.655668 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 06:49:27.656059 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260809 06:49:27.657771 19239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:27.745775 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 19072
I20260809 06:49:27.768618 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 19249
I20260809 06:49:27.785598 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:27.769633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.27.209.195:45389)
0809 06:49:27.770195 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:27.770209 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:27.770250 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:49:27.771940 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:27.771950 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:27.772386 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:27.772664 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:27.774006 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:27.775196 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:27.776493 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:27.777139 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:27.777269 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:27.779755 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:27.779766 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:27.779787 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:27.779842 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:27.782694 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:27.783321 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:27.783327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:27.783333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:27.783489 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:27.783909 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:27.783912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:27.783914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:27.784423 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:27.784653 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:27.784926 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 06:49:27.785282 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":249,"threads_started":1}
I20260809 06:49:27.787019 19442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:27.873867 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 19249
2026-08-09T06:49:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24191 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:49:27 dist-test-slave-jpj1 krb5kdc[19537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:49:27 dist-test-slave-jpj1 krb5kdc[19537](info): set up 2 sockets
Aug 09 06:49:27 dist-test-slave-jpj1 krb5kdc[19537](info): commencing operation
krb5kdc: starting...
W20260809 06:49:29.076277 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:49:29 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:49:29.338779 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36299
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:29.680572 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:29.680863 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:29.680960 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:29.688062 19556 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 06:49:29.688158 19556 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 06:49:29.688201 19556 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 06:49:29.688253 19556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:49:29.688306 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:29.688373 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:29.688418 19556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:49:29.688470 19556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:49:29.698212 19556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:36299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:29.699445 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:29.700789 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:29.712575 19563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:29.712670 19566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:29.712643 19562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:29.713408 19556 server_base.cc:1034] running on GCE node
I20260809 06:49:29.714149 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:29.715433 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:29.716641 19556 hybrid_clock.cc:648] HybridClock initialized: now 1786258169716556 us; error 56 us; skew 500 ppm
I20260809 06:49:29.716955 19556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:29 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:29.719888 19556 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:49:29.721206 19556 webserver.cc:460] Webserver started at http://127.27.209.254:44565/ using document root <none> and password file <none>
I20260809 06:49:29.721870 19556 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:29.721971 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:29.722270 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:29.725250 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "95544c1c3fc342329d592b81f88b91ff"
format_stamp: "Formatted at 2026-08-09 06:49:29 on dist-test-slave-jpj1"
server_key: "d45d8caf8f57366e7e0ca9cbc7c11f9a"
server_key_iv: "cfc0b7a75684ee9ead1d0ed4ed7b6ea0"
server_key_version: "encryptionkey@0"
I20260809 06:49:29.726004 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "95544c1c3fc342329d592b81f88b91ff"
format_stamp: "Formatted at 2026-08-09 06:49:29 on dist-test-slave-jpj1"
server_key: "d45d8caf8f57366e7e0ca9cbc7c11f9a"
server_key_iv: "cfc0b7a75684ee9ead1d0ed4ed7b6ea0"
server_key_version: "encryptionkey@0"
I20260809 06:49:29.730952 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:49:29.734297 19578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:29.735567 19556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:29.735714 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "95544c1c3fc342329d592b81f88b91ff"
format_stamp: "Formatted at 2026-08-09 06:49:29 on dist-test-slave-jpj1"
server_key: "d45d8caf8f57366e7e0ca9cbc7c11f9a"
server_key_iv: "cfc0b7a75684ee9ead1d0ed4ed7b6ea0"
server_key_version: "encryptionkey@0"
I20260809 06:49:29.735917 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:29.768859 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:29.772943 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:29.773252 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:29.790863 19556 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:36299
I20260809 06:49:29.790874 19651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:36299 every 8 connection(s)
I20260809 06:49:29.792173 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:49:29.795928 19652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:29.795948 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 19556
I20260809 06:49:29.796144 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:49:29.796852 28487 external_mini_cluster.cc:1428] Setting key fe77a685a57d1c44542683e1edeb35b0
I20260809 06:49:29.809932 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap starting.
Aug 09 06:49:29 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:29.813087 19652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:29.814534 19652 log.cc:824] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:29.817960 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: No bootstrap required, opened a new log
I20260809 06:49:29.821314 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:29.798867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.27.209.254:36299)
0809 06:49:29.799710 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:29.799725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:29.799779 (+ 54us) server_negotiation.cc:407] Connection header received
0809 06:49:29.802270 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:29.802301 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:29.802794 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:29.803255 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:29.804655 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:29.806641 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:29.807989 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:29.809020 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:29.809220 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:29.813234 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:29.813247 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:29.813266 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:29.813319 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:29.817096 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:29.817755 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:29.817764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:29.817772 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:29.817999 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:29.818514 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:29.818518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:29.818520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:29.819074 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:29.819305 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:29.819864 (+ 559us) server_negotiation.cc:299] Negotiation successful
0809 06:49:29.820259 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":315,"threads_started":1}
I20260809 06:49:29.826310 19652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:29.826676 19652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:29.826776 19652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95544c1c3fc342329d592b81f88b91ff, State: Initialized, Role: FOLLOWER
I20260809 06:49:29.827308 19652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:29.827548 19652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:49:29.827687 19652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:49:29.827844 19652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:29.830217 19652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:29.830752 19652 leader_election.cc:302] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95544c1c3fc342329d592b81f88b91ff; no voters:
I20260809 06:49:29.831270 19652 leader_election.cc:288] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:49:29.831395 19664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:29.832031 19664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 LEADER]: Becoming Leader. State: Replica: 95544c1c3fc342329d592b81f88b91ff, State: Running, Role: LEADER
I20260809 06:49:29.832540 19664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:29.833315 19652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:49:29.835116 19665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95544c1c3fc342329d592b81f88b91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } } }
I20260809 06:49:29.835426 19665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:29.835636 19667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95544c1c3fc342329d592b81f88b91ff. Latest consensus state: current_term: 1 leader_uuid: "95544c1c3fc342329d592b81f88b91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } } }
I20260809 06:49:29.835974 19667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:29.837854 19670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:49:29.843874 19670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:49:29.852699 19670 catalog_manager.cc:1347] Generated new cluster ID: f222d4b0185247a89188a8b4b06797c7
I20260809 06:49:29.852795 19670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:49:29.863350 19670 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:49:29.865067 19670 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:49:29.874133 19670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Generated new TSK 0
I20260809 06:49:29.874866 19670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:49:29.944916 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36299
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:30.282824 19691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:30.283099 19691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:30.283241 19691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:30.290340 19691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:30.290444 19691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:30.290592 19691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:49:30.300272 19691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:30.301523 19691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:30.302848 19691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:30.315697 19696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:30.315728 19697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:30.315658 19699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:30.316345 19691 server_base.cc:1034] running on GCE node
I20260809 06:49:30.316979 19691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:30.317905 19691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:30.319137 19691 hybrid_clock.cc:648] HybridClock initialized: now 1786258170319060 us; error 47 us; skew 500 ppm
I20260809 06:49:30.319453 19691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:30 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:30.322750 19691 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:49:30.324182 19691 webserver.cc:460] Webserver started at http://127.27.209.193:45963/ using document root <none> and password file <none>
I20260809 06:49:30.324878 19691 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:30.325001 19691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:30.325307 19691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:30.328261 19691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "40088212fd5449848eb22ed6ae9fd6a2"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "e671eec09d7fd20ec4d1811e810d1b9c"
server_key_iv: "c0530e04646d5859cda76ca050ff73a2"
server_key_version: "encryptionkey@0"
I20260809 06:49:30.328991 19691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "40088212fd5449848eb22ed6ae9fd6a2"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "e671eec09d7fd20ec4d1811e810d1b9c"
server_key_iv: "c0530e04646d5859cda76ca050ff73a2"
server_key_version: "encryptionkey@0"
I20260809 06:49:30.334219 19691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:49:30.337921 19708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:30.339473 19691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:30.339664 19691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "40088212fd5449848eb22ed6ae9fd6a2"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "e671eec09d7fd20ec4d1811e810d1b9c"
server_key_iv: "c0530e04646d5859cda76ca050ff73a2"
server_key_version: "encryptionkey@0"
I20260809 06:49:30.339901 19691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:30.364576 19691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:30.368467 19691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:30.368736 19691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:30.369742 19691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:30.371074 19691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:30.371152 19691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:30.371276 19691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:30.371340 19691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:30.400856 19691 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:40579
I20260809 06:49:30.400908 19877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:40579 every 8 connection(s)
I20260809 06:49:30.402460 19691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:49:30.412878 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 19691
I20260809 06:49:30.413103 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:49:30.414005 28487 external_mini_cluster.cc:1428] Setting key cc5bc4eab755f824eefbab34ab2731b6
Aug 09 06:49:30 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:30.428964 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:30.405521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:37943 (local address 127.27.209.254:36299)
0809 06:49:30.406349 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:30.406356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:30.407299 (+ 943us) server_negotiation.cc:407] Connection header received
0809 06:49:30.409545 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:30.409551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:30.409657 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:30.409724 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:30.412105 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.413166 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.415358 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.415883 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.415973 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:30.420638 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:30.420645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:30.420650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:30.420692 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:30.425360 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:30.426240 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:30.426246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:30.426250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:30.426404 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:30.426912 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:30.426918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:30.426921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:30.427252 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:30.427399 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:30.428382 (+ 983us) server_negotiation.cc:299] Negotiation successful
0809 06:49:30.428609 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":530,"threads_started":1}
I20260809 06:49:30.431478 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:30.406239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:37943)
0809 06:49:30.407114 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:30.407172 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:30.409232 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:30.410192 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:30.410202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:30.410969 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:30.411912 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:30.411938 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.413369 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.413373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:30.415139 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:30.415157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.416370 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.416378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:30.416683 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:30.417461 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:30.417492 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:30.420392 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:30.425526 (+ 5134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:30.425535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:30.425555 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:30.426108 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:30.426540 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:30.426544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:30.426547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:30.426787 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:30.427497 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:30.427505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:30.427950 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 06:49:30.429514 (+ 1564us) client_negotiation.cc:240] Negotiation successful
0809 06:49:30.430098 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":339,"threads_started":1}
I20260809 06:49:30.433643 19878 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36299
I20260809 06:49:30.433954 19878 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:30.434736 19878 heartbeater.cc:511] Master 127.27.209.254:36299 requested a full tablet report, sending...
I20260809 06:49:30.437315 19600 ts_manager.cc:194] Registered new tserver with Master: 40088212fd5449848eb22ed6ae9fd6a2 (127.27.209.193:40579)
I20260809 06:49:30.439365 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:37943
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:49:30.480526 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36299
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:30.810390 19886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:30.810664 19886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:30.810767 19886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:30.817960 19886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:30.818070 19886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:30.818189 19886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:49:30.827724 19886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:30.828922 19886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:30.830204 19886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:30.843973 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:30.844236 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:30.845348 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:30.845868 19886 server_base.cc:1034] running on GCE node
I20260809 06:49:30.846421 19886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:30.847309 19886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:30.848522 19886 hybrid_clock.cc:648] HybridClock initialized: now 1786258170848435 us; error 78 us; skew 500 ppm
I20260809 06:49:30.848853 19886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:30 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:30.851970 19886 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:49:30.853394 19886 webserver.cc:460] Webserver started at http://127.27.209.194:33171/ using document root <none> and password file <none>
I20260809 06:49:30.854099 19886 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:30.854218 19886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:30.854512 19886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:30.857419 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "b8a364b2ba4149378e9c3343802b3fba"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "149628ea9438162c73aa990d853b5046"
server_key_iv: "b64262517961511fd8ca2e2d7d19d80f"
server_key_version: "encryptionkey@0"
I20260809 06:49:30.858153 19886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "b8a364b2ba4149378e9c3343802b3fba"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "149628ea9438162c73aa990d853b5046"
server_key_iv: "b64262517961511fd8ca2e2d7d19d80f"
server_key_version: "encryptionkey@0"
I20260809 06:49:30.863135 19886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:49:30.866420 19905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:30.867667 19886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:49:30.867815 19886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "b8a364b2ba4149378e9c3343802b3fba"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "149628ea9438162c73aa990d853b5046"
server_key_iv: "b64262517961511fd8ca2e2d7d19d80f"
server_key_version: "encryptionkey@0"
I20260809 06:49:30.867990 19886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:30.886246 19886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:30.890014 19886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:30.890283 19886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:30.891160 19886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:30.892481 19886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:30.892561 19886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:30.892683 19886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:30.892743 19886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:30.924324 19886 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:40831
I20260809 06:49:30.924414 20083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:40831 every 8 connection(s)
I20260809 06:49:30.925657 19886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:49:30.935286 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 19886
I20260809 06:49:30.935495 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:49:30.936106 28487 external_mini_cluster.cc:1428] Setting key 3ebc02c0be123c065980b327af117a6c
Aug 09 06:49:30 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:30.946420 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:30.928147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:42593 (local address 127.27.209.254:36299)
0809 06:49:30.928379 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:30.928385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:30.929781 (+ 1396us) server_negotiation.cc:407] Connection header received
0809 06:49:30.931580 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:30.931586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:30.931662 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:30.931756 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:30.933323 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.934068 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.936003 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.936476 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.936572 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:30.940450 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:30.940457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:30.940461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:30.940498 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:30.943441 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:30.944180 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:30.944185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:30.944188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:30.944348 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:30.944772 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:30.944776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:30.944778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:30.945111 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:30.945254 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:30.945894 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 06:49:30.946109 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:49:30.947261 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:30.928865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:42593)
0809 06:49:30.929621 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:30.929661 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:30.931311 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:30.931978 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:30.931986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:30.932541 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:30.933153 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:30.933173 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.934212 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.934216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:30.935866 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:30.935882 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:30.936707 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:30.936714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:30.936962 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:30.937664 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:30.937692 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:30.940209 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:30.943604 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:30.943611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:30.943623 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:30.944064 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:30.944446 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:30.944450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:30.944452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:30.944666 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:30.945215 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:30.945221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:30.945532 (+ 311us) client_negotiation.cc:769] Sending connection context
0809 06:49:30.945761 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 06:49:30.946134 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":292,"threads_started":1}
I20260809 06:49:30.948697 20084 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36299
I20260809 06:49:30.948977 20084 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:30.949707 20084 heartbeater.cc:511] Master 127.27.209.254:36299 requested a full tablet report, sending...
I20260809 06:49:30.951351 19600 ts_manager.cc:194] Registered new tserver with Master: b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831)
I20260809 06:49:30.952438 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:42593
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:49:30.997851 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36299
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:31.336925 20089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:31.337217 20089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:31.337323 20089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:31.344998 20089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:31.345124 20089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:31.345248 20089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:49:31.355257 20089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:31.356627 20089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:31.357945 20089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:31.371300 20098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:31.371271 20099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:31.371341 20101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:31.372542 20089 server_base.cc:1034] running on GCE node
I20260809 06:49:31.373153 20089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:31.374066 20089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:31.375277 20089 hybrid_clock.cc:648] HybridClock initialized: now 1786258171375211 us; error 45 us; skew 500 ppm
I20260809 06:49:31.375612 20089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:31 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:31.378691 20089 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:49:31.380056 20089 webserver.cc:460] Webserver started at http://127.27.209.195:45043/ using document root <none> and password file <none>
I20260809 06:49:31.380707 20089 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:31.380803 20089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:31.381054 20089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:49:31.383862 20089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "28dec15678af4232ad692b3c86d092b7"
format_stamp: "Formatted at 2026-08-09 06:49:31 on dist-test-slave-jpj1"
server_key: "ca7e920fba62cdfe1874eabf61c8c612"
server_key_iv: "b3391cb35de660bb6c786647323611c2"
server_key_version: "encryptionkey@0"
I20260809 06:49:31.384565 20089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "28dec15678af4232ad692b3c86d092b7"
format_stamp: "Formatted at 2026-08-09 06:49:31 on dist-test-slave-jpj1"
server_key: "ca7e920fba62cdfe1874eabf61c8c612"
server_key_iv: "b3391cb35de660bb6c786647323611c2"
server_key_version: "encryptionkey@0"
I20260809 06:49:31.389477 20089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:49:31.392844 20112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:31.394074 20089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:49:31.394210 20089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "28dec15678af4232ad692b3c86d092b7"
format_stamp: "Formatted at 2026-08-09 06:49:31 on dist-test-slave-jpj1"
server_key: "ca7e920fba62cdfe1874eabf61c8c612"
server_key_iv: "b3391cb35de660bb6c786647323611c2"
server_key_version: "encryptionkey@0"
I20260809 06:49:31.394373 20089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:31.436774 20089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:31.441247 20089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:31.441535 20089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:31.442495 20089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:31.443607 19878 heartbeater.cc:503] Master 127.27.209.254:36299 was elected leader, sending a full tablet report...
I20260809 06:49:31.444046 20089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:31.444183 20089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 06:49:31.444347 20089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:31.444412 20089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:31.477120 20089 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:44013
I20260809 06:49:31.477166 20269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:44013 every 8 connection(s)
I20260809 06:49:31.478348 20089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:49:31.483794 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 20089
I20260809 06:49:31.483956 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:49:31.484407 28487 external_mini_cluster.cc:1428] Setting key e054b8259048e7d4325ec0954be2ec38
Aug 09 06:49:31 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:31.497473 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.480912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:40653 (local address 127.27.209.254:36299)
0809 06:49:31.481347 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.481352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.482305 (+ 953us) server_negotiation.cc:407] Connection header received
0809 06:49:31.484177 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.484182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.484266 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.484365 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:31.486089 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.487002 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.488661 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.489041 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.489142 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.492277 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:31.492281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:31.492285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:31.492316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:31.494600 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.495267 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.495271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.495273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.495396 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.495834 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.495838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.495840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.496092 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:31.496234 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.496890 (+ 656us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.497105 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260809 06:49:31.498435 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.481417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:40653)
0809 06:49:31.482123 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:31.482179 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:31.483873 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:31.484649 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:31.484659 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:31.485289 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:31.485930 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.485950 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.487159 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.487165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.488518 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.488531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.489223 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.489228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.489423 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.489991 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:31.490013 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:31.492088 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:31.494726 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:31.494734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:31.494750 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:31.495149 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:31.495495 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:31.495500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:31.495504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:31.495722 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:31.496226 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:31.496234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:31.496532 (+ 298us) client_negotiation.cc:769] Sending connection context
0809 06:49:31.496761 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 06:49:31.497237 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":278,"threads_started":1}
I20260809 06:49:31.499902 20270 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36299
I20260809 06:49:31.500195 20270 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:31.500878 20270 heartbeater.cc:511] Master 127.27.209.254:36299 requested a full tablet report, sending...
I20260809 06:49:31.502202 19599 ts_manager.cc:194] Registered new tserver with Master: 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
I20260809 06:49:31.503055 19599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:40653
I20260809 06:49:31.511746 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:49:31.528935 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.516251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.27.209.254:36299)
0809 06:49:31.516437 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.516444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.516906 (+ 462us) server_negotiation.cc:407] Connection header received
0809 06:49:31.517030 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.517034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.517092 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.517163 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:31.518666 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.519613 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.520990 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.521659 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.521751 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.524004 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:31.524009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:31.524012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:31.524042 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:31.526722 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.527296 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.527299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.527302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.527411 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.527721 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.527724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.527726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.528054 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:31.528238 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.528518 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.528679 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:49:31.532855 19599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:49:31.539712 19599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:49:31.567426 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.552282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:41110)
0809 06:49:31.555933 (+ 3651us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:31.555956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:31.556184 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:31.557181 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:31.557190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:31.557223 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:31.557630 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.557649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.560925 (+ 3276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.560932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.563499 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.563516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.566600 (+ 3084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.566607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.566825 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.566875 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 06:49:31.567008 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 06:49:31.567163 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2380,"thread_start_us":2150,"threads_started":1}
I20260809 06:49:31.567621 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.552581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.27.209.193:40579)
0809 06:49:31.553377 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.553388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.556053 (+ 2665us) server_negotiation.cc:407] Connection header received
0809 06:49:31.556384 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.556393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.556667 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.556858 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:31.557830 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.559602 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.563720 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.565391 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.565566 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.565749 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.567090 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.567271 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":418,"threads_started":1}
I20260809 06:49:31.571731 19781 tablet_service.cc:1467] Processing CreateTablet for tablet b77febabb3cf45e5ac298211bdf9abd7 (DEFAULT_TABLE table=test-table [id=174672e78e3c4533b100575f9fabc50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:31.573357 19781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77febabb3cf45e5ac298211bdf9abd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:31.574375 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.554677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40831 (local address 127.0.0.1:47274)
0809 06:49:31.566448 (+ 11771us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:31.566468 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:31.567747 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:31.568450 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:31.568453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:31.568472 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:31.568849 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.568871 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.570266 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.570271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.572091 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.572106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.573633 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.573640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.573788 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.573810 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 06:49:31.573891 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:49:31.574011 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11632,"mutex_wait_us":1,"thread_start_us":11477,"threads_started":1}
I20260809 06:49:31.574414 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.555058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.27.209.194:40831)
0809 06:49:31.565805 (+ 10747us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.565813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.566570 (+ 757us) server_negotiation.cc:407] Connection header received
0809 06:49:31.567948 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.567954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.568116 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.568247 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:31.569044 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.570144 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.572234 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.573358 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.573504 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.573671 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.573954 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.574105 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10564,"thread_start_us":10738,"threads_started":1}
I20260809 06:49:31.575770 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.552020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44013 (local address 127.0.0.1:37994)
0809 06:49:31.553898 (+ 1878us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:31.553938 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:31.554139 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:31.568740 (+ 14601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:31.568745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:31.568773 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:31.569185 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.569199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.571091 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.571096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.573153 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.573169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.574950 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.574956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.575120 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.575150 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:49:31.575241 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 06:49:31.575358 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1628,"mutex_wait_us":27,"thread_start_us":1502,"threads_started":1}
I20260809 06:49:31.575837 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.552294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.27.209.195:44013)
0809 06:49:31.567072 (+ 14778us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.567079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.567110 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:49:31.567204 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.567212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.567469 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.568737 (+ 1268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:31.569400 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.570924 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.573345 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.574713 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.574953 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.575104 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.575291 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.575458 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14615,"thread_start_us":14465,"threads_started":1}
I20260809 06:49:31.578305 19983 tablet_service.cc:1467] Processing CreateTablet for tablet b77febabb3cf45e5ac298211bdf9abd7 (DEFAULT_TABLE table=test-table [id=174672e78e3c4533b100575f9fabc50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:31.579393 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77febabb3cf45e5ac298211bdf9abd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:31.579272 20179 tablet_service.cc:1467] Processing CreateTablet for tablet b77febabb3cf45e5ac298211bdf9abd7 (DEFAULT_TABLE table=test-table [id=174672e78e3c4533b100575f9fabc50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:31.580928 20179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b77febabb3cf45e5ac298211bdf9abd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:31.588804 20299 tablet_bootstrap.cc:492] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Bootstrap starting.
I20260809 06:49:31.591734 20301 tablet_bootstrap.cc:492] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Bootstrap starting.
I20260809 06:49:31.592878 20299 tablet_bootstrap.cc:654] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:31.594337 20299 log.cc:824] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:31.594439 20302 tablet_bootstrap.cc:492] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Bootstrap starting.
I20260809 06:49:31.595703 20301 tablet_bootstrap.cc:654] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:31.596676 20301 log.cc:824] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:31.597800 20302 tablet_bootstrap.cc:654] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:31.597898 20299 tablet_bootstrap.cc:492] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: No bootstrap required, opened a new log
I20260809 06:49:31.598201 20299 ts_tablet_manager.cc:1397] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 06:49:31.599062 20302 log.cc:824] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:31.599255 20301 tablet_bootstrap.cc:492] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: No bootstrap required, opened a new log
I20260809 06:49:31.599494 20301 ts_tablet_manager.cc:1397] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260809 06:49:31.602125 20302 tablet_bootstrap.cc:492] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: No bootstrap required, opened a new log
I20260809 06:49:31.602392 20302 ts_tablet_manager.cc:1397] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260809 06:49:31.606827 20301 raft_consensus.cc:348] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.607184 20301 raft_consensus.cc:374] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:31.607285 20301 raft_consensus.cc:729] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a364b2ba4149378e9c3343802b3fba, State: Initialized, Role: FOLLOWER
I20260809 06:49:31.607867 20301 consensus_queue.cc:260] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.608249 20299 raft_consensus.cc:348] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.608656 20299 raft_consensus.cc:374] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:31.608760 20299 raft_consensus.cc:729] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40088212fd5449848eb22ed6ae9fd6a2, State: Initialized, Role: FOLLOWER
I20260809 06:49:31.608996 20084 heartbeater.cc:503] Master 127.27.209.254:36299 was elected leader, sending a full tablet report...
I20260809 06:49:31.609354 20299 consensus_queue.cc:260] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.611131 20299 ts_tablet_manager.cc:1428] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260809 06:49:31.612432 20301 ts_tablet_manager.cc:1428] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20260809 06:49:31.612668 20302 raft_consensus.cc:348] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.613047 20302 raft_consensus.cc:374] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:31.613150 20302 raft_consensus.cc:729] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dec15678af4232ad692b3c86d092b7, State: Initialized, Role: FOLLOWER
I20260809 06:49:31.613708 20302 consensus_queue.cc:260] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.614897 20270 heartbeater.cc:503] Master 127.27.209.254:36299 was elected leader, sending a full tablet report...
I20260809 06:49:31.615398 20302 ts_tablet_manager.cc:1428] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260809 06:49:31.655298 19879 tablet.cc:2369] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:31.657011 20309 raft_consensus.cc:482] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:49:31.657289 20309 raft_consensus.cc:504] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.658902 20309 leader_election.cc:288] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013), b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831)
I20260809 06:49:31.665160 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.659251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:50481 (local address 127.27.209.195:44013)
0809 06:49:31.659449 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.659455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.660150 (+ 695us) server_negotiation.cc:407] Connection header received
0809 06:49:31.660353 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.660357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.660421 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.660495 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:31.661060 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.661773 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.663509 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.664306 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.664450 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.664523 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.664803 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.664923 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:49:31.665331 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.659375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:45555 (local address 127.27.209.194:40831)
0809 06:49:31.659601 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.659606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.660531 (+ 925us) server_negotiation.cc:407] Connection header received
0809 06:49:31.660691 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.660695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.660755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.660827 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:31.661345 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.662044 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.663721 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.664471 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.664606 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.664676 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.664966 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.665086 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:49:31.665370 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.659199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44013 (local address 127.27.209.193:50481)
0809 06:49:31.660032 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:31.660047 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:31.660207 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:31.660530 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:31.660535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:31.660625 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:31.660912 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.660921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.661896 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.661901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.663355 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.663367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.664500 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.664506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.664649 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.664678 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 06:49:31.664761 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 06:49:31.664882 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"spinlock_wait_cycles":22784,"thread_start_us":547,"threads_started":1}
I20260809 06:49:31.665460 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.659410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40831 (local address 127.27.209.193:45555)
0809 06:49:31.660424 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:31.660439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:31.660564 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:31.660872 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:31.660876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:31.660903 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:31.661199 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.661207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.662184 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.662188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.663580 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:31.663595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.664688 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.664695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:31.664822 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.664845 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:49:31.664924 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 06:49:31.665044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":1,"thread_start_us":740,"threads_started":1}
I20260809 06:49:31.666431 20019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77febabb3cf45e5ac298211bdf9abd7" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8a364b2ba4149378e9c3343802b3fba" is_pre_election: true
I20260809 06:49:31.666430 20208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77febabb3cf45e5ac298211bdf9abd7" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dec15678af4232ad692b3c86d092b7" is_pre_election: true
I20260809 06:49:31.666991 20019 raft_consensus.cc:2454] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 0.
I20260809 06:49:31.666990 20208 raft_consensus.cc:2454] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 0.
I20260809 06:49:31.667717 19709 leader_election.cc:302] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dec15678af4232ad692b3c86d092b7, 40088212fd5449848eb22ed6ae9fd6a2; no voters:
I20260809 06:49:31.668157 20309 raft_consensus.cc:2781] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:49:31.668289 20309 raft_consensus.cc:482] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:49:31.668380 20309 raft_consensus.cc:3037] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:31.670547 20309 raft_consensus.cc:504] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
I20260809 06:49:31.671092 20309 leader_election.cc:288] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 election: Requested vote from peers 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013), b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831)
I20260809 06:49:31.671625 20208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77febabb3cf45e5ac298211bdf9abd7" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dec15678af4232ad692b3c86d092b7"
I20260809 06:49:31.671625 20019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b77febabb3cf45e5ac298211bdf9abd7" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8a364b2ba4149378e9c3343802b3fba"
I20260809 06:49:31.671841 20019 raft_consensus.cc:3037] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:31.671846 20208 raft_consensus.cc:3037] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:31.675009 20208 raft_consensus.cc:2454] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 1.
I20260809 06:49:31.675009 20019 raft_consensus.cc:2454] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 1.
I20260809 06:49:31.675462 19710 leader_election.cc:302] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2, b8a364b2ba4149378e9c3343802b3fba; no voters:
I20260809 06:49:31.676043 20309 raft_consensus.cc:2781] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:31.676611 20309 raft_consensus.cc:686] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 LEADER]: Becoming Leader. State: Replica: 40088212fd5449848eb22ed6ae9fd6a2, State: Running, Role: LEADER
I20260809 06:49:31.677114 20309 consensus_queue.cc:237] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } }
W20260809 06:49:31.677595 20085 tablet.cc:2369] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:31.681439 19599 catalog_manager.cc:5495] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40088212fd5449848eb22ed6ae9fd6a2 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "40088212fd5449848eb22ed6ae9fd6a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } health_report { overall_health: UNKNOWN } } }
I20260809 06:49:31.718422 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.713497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.27.209.193:40579)
0809 06:49:31.713679 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.713684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.713700 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:49:31.713794 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.713798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.713872 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.713990 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:49:31.714331 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.715073 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.716425 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.716910 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.717003 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.717441 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:49:31.717937 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:49:31.718049 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.718090 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.718151 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:49:31.726271 20019 raft_consensus.cc:1261] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Refusing update from remote peer 40088212fd5449848eb22ed6ae9fd6a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:31.726301 20208 raft_consensus.cc:1261] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Refusing update from remote peer 40088212fd5449848eb22ed6ae9fd6a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:31.727173 20317 consensus_queue.cc:1035] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:31.727574 20309 consensus_queue.cc:1035] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:49:31.729856 20271 tablet.cc:2369] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:31.738332 20322 mvcc.cc:205] Tried to move back new op lower bound from 7316513471381438464 to 7316513471193079808. Current Snapshot: MvccSnapshot[applied={T|T < 7316513471381438464}]
I20260809 06:49:31.741627 20324 mvcc.cc:205] Tried to move back new op lower bound from 7316513471381438464 to 7316513471193079808. Current Snapshot: MvccSnapshot[applied={T|T < 7316513471381438464}]
I20260809 06:49:31.752363 19599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:31.752605 19599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:31.755935 19599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Sending DeleteTablet for 3 replicas of tablet b77febabb3cf45e5ac298211bdf9abd7
I20260809 06:49:31.757018 20179 tablet_service.cc:1514] Processing DeleteTablet for tablet b77febabb3cf45e5ac298211bdf9abd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:31 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:37994
I20260809 06:49:31.757229 19781 tablet_service.cc:1514] Processing DeleteTablet for tablet b77febabb3cf45e5ac298211bdf9abd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:31 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:41110
I20260809 06:49:31.757200 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet b77febabb3cf45e5ac298211bdf9abd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:31 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:47274
I20260809 06:49:31.758132 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 19691
I20260809 06:49:31.768013 20333 tablet_replica.cc:330] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: stopping tablet replica
I20260809 06:49:31.768617 20333 raft_consensus.cc:2229] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:31.769310 20333 raft_consensus.cc:2258] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:31.771162 20336 tablet_replica.cc:330] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: stopping tablet replica
I20260809 06:49:31.771647 20336 raft_consensus.cc:2229] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:31.771745 20333 ts_tablet_manager.cc:1905] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:31.772274 20336 raft_consensus.cc:2258] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:31.774319 20336 ts_tablet_manager.cc:1905] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:31.776609 20335 tablet_replica.cc:330] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: stopping tablet replica
I20260809 06:49:31.778494 20333 ts_tablet_manager.cc:1918] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:31.778650 20333 log.cc:1196] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/b77febabb3cf45e5ac298211bdf9abd7
I20260809 06:49:31.779167 20333 ts_tablet_manager.cc:1939] T b77febabb3cf45e5ac298211bdf9abd7 P 28dec15678af4232ad692b3c86d092b7: Deleting consensus metadata
I20260809 06:49:31.779443 20335 raft_consensus.cc:2229] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:49:31.780205 20335 raft_consensus.cc:2258] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:31.781032 19579 catalog_manager.cc:4841] TS 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): tablet b77febabb3cf45e5ac298211bdf9abd7 (table test-table [id=174672e78e3c4533b100575f9fabc50f]) successfully deleted
Aug 09 06:49:31 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:49:31.783171 20335 ts_tablet_manager.cc:1905] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:31.784149 20336 ts_tablet_manager.cc:1918] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:31.784312 20336 log.cc:1196] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/b77febabb3cf45e5ac298211bdf9abd7
I20260809 06:49:31.784809 20336 ts_tablet_manager.cc:1939] T b77febabb3cf45e5ac298211bdf9abd7 P b8a364b2ba4149378e9c3343802b3fba: Deleting consensus metadata
I20260809 06:49:31.786584 19580 catalog_manager.cc:4841] TS b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): tablet b77febabb3cf45e5ac298211bdf9abd7 (table test-table [id=174672e78e3c4533b100575f9fabc50f]) successfully deleted
I20260809 06:49:31.790041 20335 ts_tablet_manager.cc:1918] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:31.790235 20335 log.cc:1196] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/b77febabb3cf45e5ac298211bdf9abd7
I20260809 06:49:31.790777 20335 ts_tablet_manager.cc:1939] T b77febabb3cf45e5ac298211bdf9abd7 P 40088212fd5449848eb22ed6ae9fd6a2: Deleting consensus metadata
I20260809 06:49:31.791896 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.759239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.27.209.193:40579)
0809 06:49:31.759440 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.759446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.770494 (+ 11048us) server_negotiation.cc:407] Connection header received
0809 06:49:31.770743 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.770748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.770837 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.770934 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:31.773232 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.774495 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.776958 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.777717 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.777824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.783736 (+ 5912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:31.783755 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:31.783773 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:31.783824 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:31.788048 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.788995 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.789005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.789008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.789298 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.789770 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.789775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.789779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.790246 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:31.790643 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.791260 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.791515 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:49:31.792317 19582 catalog_manager.cc:4841] TS 40088212fd5449848eb22ed6ae9fd6a2 (127.27.209.193:40579): tablet b77febabb3cf45e5ac298211bdf9abd7 (table test-table [id=174672e78e3c4533b100575f9fabc50f]) successfully deleted
I20260809 06:49:31.793354 19872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:31.891758 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 19691
I20260809 06:49:31.916620 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 19886
Aug 09 06:49:31 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:49:31.932896 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:31.917436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.27.209.194:40831)
0809 06:49:31.917605 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:31.917609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:31.917627 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:49:31.917746 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:31.917750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:31.917815 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:31.917893 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:31.919157 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.919946 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.921226 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:31.921641 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:31.921736 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:31.926107 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:31.926120 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:31.926136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:31.926178 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:31.929897 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.930562 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.930572 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.930576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.930717 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:31.931105 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:31.931110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:31.931113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:31.931700 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:31.931993 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:31.932260 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 06:49:31.932507 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:49:31.933641 20077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:32.021260 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 19886
I20260809 06:49:32.045452 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 20089
Aug 09 06:49:32 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:49:32.059952 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:32.046224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.27.209.195:44013)
0809 06:49:32.046407 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:32.046412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:32.046432 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:49:32.046589 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:32.046593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:32.046668 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:32.046752 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:32.048173 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:32.048902 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:32.050251 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:32.050632 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:32.050721 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:32.054977 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:32.054992 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:32.055009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:32.055041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:32.057506 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:32.058080 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:32.058086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:32.058089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:32.058201 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:32.058520 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:32.058523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:32.058526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:32.058889 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:32.059099 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:32.059319 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 06:49:32.059482 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 06:49:32.060549 20264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:32.148073 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 20089
I20260809 06:49:32.167876 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 19556
I20260809 06:49:32.169045 19646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:32.278530 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 19556
I20260809 06:49:52.307870 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36299
--webserver_interface=127.27.209.254
--webserver_port=44565
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:49:52.647826 20346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:52.648100 20346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:52.648223 20346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:52.655336 20346 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 06:49:52.655421 20346 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 06:49:52.655465 20346 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 06:49:52.655555 20346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:49:52.655633 20346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:52.655721 20346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:52.655777 20346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:49:52.655851 20346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:49:52.665542 20346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:36299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=44565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:52.666792 20346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:52.668177 20346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:52.692435 20359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:52.692469 20362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:52.692430 20360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:52.693238 20346 server_base.cc:1034] running on GCE node
I20260809 06:49:52.694095 20346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:52.695713 20346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:52.696959 20346 hybrid_clock.cc:648] HybridClock initialized: now 1786258192696879 us; error 48 us; skew 500 ppm
I20260809 06:49:52.697357 20346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:52 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258192, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:52.700928 20346 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:49:52.702579 20346 webserver.cc:460] Webserver started at http://127.27.209.254:44565/ using document root <none> and password file <none>
I20260809 06:49:52.703331 20346 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:52.703449 20346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:52.709422 20346 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:49:52.712427 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:52.713799 20346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:49:52.713955 20346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "95544c1c3fc342329d592b81f88b91ff"
format_stamp: "Formatted at 2026-08-09 06:49:29 on dist-test-slave-jpj1"
server_key: "d45d8caf8f57366e7e0ca9cbc7c11f9a"
server_key_iv: "cfc0b7a75684ee9ead1d0ed4ed7b6ea0"
server_key_version: "encryptionkey@0"
I20260809 06:49:52.715013 20346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:52.738802 20346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:52.742965 20346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:52.743256 20346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:52.761940 20346 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:36299
I20260809 06:49:52.761971 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:36299 every 8 connection(s)
I20260809 06:49:52.763425 20346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:49:52.767038 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 20346
I20260809 06:49:52.768206 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:40579
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:36299
--webserver_port=45963
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:49:52.772112 20444 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:49:52.776551 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap starting.
I20260809 06:49:52.797043 20444 log.cc:824] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:52.809369 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:49:52.809928 20444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap complete.
I20260809 06:49:52.819500 20444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:52.820457 20444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95544c1c3fc342329d592b81f88b91ff, State: Initialized, Role: FOLLOWER
I20260809 06:49:52.821132 20444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:52.821359 20444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:49:52.821475 20444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:49:52.821635 20444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 1 FOLLOWER]: Advancing to term 2
I20260809 06:49:52.825426 20444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:52.825940 20444 leader_election.cc:302] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95544c1c3fc342329d592b81f88b91ff; no voters:
I20260809 06:49:52.826527 20444 leader_election.cc:288] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 06:49:52.826655 20451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Leader election won for term 2
I20260809 06:49:52.827760 20451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 LEADER]: Becoming Leader. State: Replica: 95544c1c3fc342329d592b81f88b91ff, State: Running, Role: LEADER
I20260809 06:49:52.828310 20451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:49:52.828815 20444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:49:52.830921 20456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95544c1c3fc342329d592b81f88b91ff. Latest consensus state: current_term: 2 leader_uuid: "95544c1c3fc342329d592b81f88b91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } } }
I20260809 06:49:52.830952 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95544c1c3fc342329d592b81f88b91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } } }
I20260809 06:49:52.831344 20456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:52.831348 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: This master's current role is: LEADER
I20260809 06:49:52.838912 20462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:49:52.847424 20462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:49:52.849568 20462 catalog_manager.cc:1259] Loaded cluster ID: f222d4b0185247a89188a8b4b06797c7
I20260809 06:49:52.849689 20462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:49:52.854369 20462 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:49:52.856020 20462 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Loaded TSK: 0
I20260809 06:49:52.871289 20476 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:49:52.883219 20462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Generated new TSK 1
I20260809 06:49:52.883638 20462 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:49:53.142591 20446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:53.142896 20446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:53.143033 20446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:53.150374 20446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:53.150503 20446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:53.150660 20446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:49:53.160462 20446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=45963
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:53.161761 20446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:53.163096 20446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:53.178099 20486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:53.178126 20484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:53.179975 20482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:53.181116 20446 server_base.cc:1034] running on GCE node
I20260809 06:49:53.181757 20446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:53.182734 20446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:53.183945 20446 hybrid_clock.cc:648] HybridClock initialized: now 1786258193183893 us; error 44 us; skew 500 ppm
I20260809 06:49:53.184262 20446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:53 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258193, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:53.187353 20446 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:49:53.188910 20446 webserver.cc:460] Webserver started at http://127.27.209.193:45963/ using document root <none> and password file <none>
I20260809 06:49:53.189592 20446 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:53.189725 20446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:53.195099 20446 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:49:53.197821 20502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:53.199153 20446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 06:49:53.199292 20446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "40088212fd5449848eb22ed6ae9fd6a2"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "e671eec09d7fd20ec4d1811e810d1b9c"
server_key_iv: "c0530e04646d5859cda76ca050ff73a2"
server_key_version: "encryptionkey@0"
I20260809 06:49:53.200270 20446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:53.221935 20446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:53.226020 20446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:53.226269 20446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:53.227202 20446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:53.228533 20446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:53.228627 20446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:53.228746 20446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:53.228821 20446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:53.260051 20446 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:40579
I20260809 06:49:53.260124 20658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:40579 every 8 connection(s)
I20260809 06:49:53.261523 20446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:49:53.270426 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 20446
I20260809 06:49:53.271049 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:40831
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:36299
--webserver_port=33171
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:49:53 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258193, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:53.285755 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:53.264709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:42195)
0809 06:49:53.265526 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:53.265568 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:53.267317 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:53.268733 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:53.268743 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:53.269358 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:53.269930 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:53.269957 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.271751 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.271757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:53.273551 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:53.273568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.274621 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.274627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:53.274886 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:53.275653 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:53.275688 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:53.278333 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:53.281751 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:53.281759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:53.281774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:53.282228 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:53.282774 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:53.282777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:53.282779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:53.282950 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:53.283852 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:53.283858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:53.284158 (+ 300us) client_negotiation.cc:769] Sending connection context
0809 06:49:53.284392 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 06:49:53.284767 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":376,"threads_started":1}
I20260809 06:49:53.286182 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:53.264445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:42195 (local address 127.27.209.254:36299)
0809 06:49:53.265016 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:53.265028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:53.265724 (+ 696us) server_negotiation.cc:407] Connection header received
0809 06:49:53.267884 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:53.267899 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:53.268216 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:53.268552 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:53.270163 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.271528 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.273716 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.274377 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.274517 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:53.278628 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:53.278637 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:53.278652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:53.278694 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:53.281609 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:53.282382 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:53.282392 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:53.282400 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:53.282627 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:53.283051 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:53.283056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:53.283060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:53.283710 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:53.283893 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:53.284633 (+ 740us) server_negotiation.cc:299] Negotiation successful
0809 06:49:53.285051 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260809 06:49:53.288940 20659 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36299
I20260809 06:49:53.289222 20659 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:53.289932 20659 heartbeater.cc:511] Master 127.27.209.254:36299 requested a full tablet report, sending...
I20260809 06:49:53.292121 20396 ts_manager.cc:194] Registered new tserver with Master: 40088212fd5449848eb22ed6ae9fd6a2 (127.27.209.193:40579)
I20260809 06:49:53.293586 20396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:42195
W20260809 06:49:53.602687 20665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:53.602963 20665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:53.603097 20665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:53.610275 20665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:53.610381 20665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:53.610491 20665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:49:53.620127 20665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:40831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=33171
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:53.621423 20665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:53.622701 20665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:53.635921 20671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:53.635841 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:53.635885 20672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:53.637018 20665 server_base.cc:1034] running on GCE node
I20260809 06:49:53.637674 20665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:53.638649 20665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:53.639869 20665 hybrid_clock.cc:648] HybridClock initialized: now 1786258193639810 us; error 41 us; skew 500 ppm
I20260809 06:49:53.640192 20665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:53 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258193, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:53.643190 20665 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:49:53.644579 20665 webserver.cc:460] Webserver started at http://127.27.209.194:33171/ using document root <none> and password file <none>
I20260809 06:49:53.645260 20665 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:53.645390 20665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:53.650735 20665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 06:49:53.653700 20685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:53.655021 20665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:53.655187 20665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "b8a364b2ba4149378e9c3343802b3fba"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "149628ea9438162c73aa990d853b5046"
server_key_iv: "b64262517961511fd8ca2e2d7d19d80f"
server_key_version: "encryptionkey@0"
I20260809 06:49:53.656177 20665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:53.679658 20665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:53.683578 20665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:53.683842 20665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:53.684865 20665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:53.686277 20665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:53.686385 20665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:53.686497 20665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:53.686568 20665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:53.717154 20665 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:40831
I20260809 06:49:53.717223 20841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:40831 every 8 connection(s)
I20260809 06:49:53.718600 20665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:49:53.721046 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 20665
I20260809 06:49:53.721731 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:44013
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:36299
--webserver_port=45043
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 06:49:53 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258193, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:53.739099 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:53.721283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:32923 (local address 127.27.209.254:36299)
0809 06:49:53.721504 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:53.721510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:53.722711 (+ 1201us) server_negotiation.cc:407] Connection header received
0809 06:49:53.724596 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:53.724602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:53.724685 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:53.724796 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:53.726196 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.727114 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.728817 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.729413 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.729535 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:53.733132 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:53.733137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:53.733141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:53.733175 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:53.736016 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:53.736788 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:53.736794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:53.736797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:53.736957 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:53.737391 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:53.737395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:53.737398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:53.737728 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:53.737872 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:53.738548 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 06:49:53.738779 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:49:53.739959 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:53.721691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:32923)
0809 06:49:53.722501 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:53.722555 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:53.724321 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:53.724978 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:53.724985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:53.725466 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:53.726017 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:53.726034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.727264 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.727268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:53.728672 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:53.728684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:53.729643 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:53.729649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:53.729842 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:53.730506 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:53.730528 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:53.732870 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:53.736169 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:53.736176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:53.736190 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:53.736619 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:53.737067 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:53.737071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:53.737073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:53.737270 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:53.737845 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:53.737851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:53.738164 (+ 313us) client_negotiation.cc:769] Sending connection context
0809 06:49:53.738400 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 06:49:53.738834 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":350,"threads_started":1}
I20260809 06:49:53.741475 20842 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36299
I20260809 06:49:53.741770 20842 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:53.742456 20842 heartbeater.cc:511] Master 127.27.209.254:36299 requested a full tablet report, sending...
I20260809 06:49:53.744059 20396 ts_manager.cc:194] Registered new tserver with Master: b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831)
I20260809 06:49:53.745124 20396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:32923
W20260809 06:49:54.052148 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:49:54.052438 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:49:54.052534 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:49:54.059507 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:49:54.059666 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:49:54.059793 20847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:49:54.069191 20847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=45043
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:49:54.070480 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:49:54.071782 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:49:54.084286 20852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:54.084388 20853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:49:54.084561 20856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:49:54.085403 20847 server_base.cc:1034] running on GCE node
I20260809 06:49:54.085956 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:49:54.086838 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:49:54.088016 20847 hybrid_clock.cc:648] HybridClock initialized: now 1786258194087962 us; error 51 us; skew 500 ppm
I20260809 06:49:54.088313 20847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:49:54 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258194, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:49:54.091292 20847 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:49:54.092733 20847 webserver.cc:460] Webserver started at http://127.27.209.195:45043/ using document root <none> and password file <none>
I20260809 06:49:54.093394 20847 fs_manager.cc:359] Metadata directory not provided
I20260809 06:49:54.093528 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:49:54.098850 20847 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 06:49:54.101676 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:54.102965 20847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:49:54.103122 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "28dec15678af4232ad692b3c86d092b7"
format_stamp: "Formatted at 2026-08-09 06:49:31 on dist-test-slave-jpj1"
server_key: "ca7e920fba62cdfe1874eabf61c8c612"
server_key_iv: "b3391cb35de660bb6c786647323611c2"
server_key_version: "encryptionkey@0"
I20260809 06:49:54.104130 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:49:54.120136 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:49:54.123999 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:49:54.124238 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:49:54.125169 20847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:49:54.126430 20847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:49:54.126525 20847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:54.126634 20847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:49:54.126714 20847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:49:54.158986 20847 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:44013
I20260809 06:49:54.159036 21035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:44013 every 8 connection(s)
I20260809 06:49:54.160543 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:49:54.161916 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 20847
Aug 09 06:49:54 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258194, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:49:54.177978 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.163000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.27.209.254:36299)
0809 06:49:54.163402 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.163408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.163685 (+ 277us) server_negotiation.cc:407] Connection header received
0809 06:49:54.163951 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.163955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.164020 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.164103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:54.166010 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.167130 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.168542 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.169114 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.169219 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.172225 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:54.172232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:54.172236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:54.172272 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:54.175166 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.175983 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.175987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.175989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.176133 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.176532 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.176536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.176538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.176857 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:54.177122 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.177429 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.177635 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":285}
I20260809 06:49:54.181458 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.163122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:59921 (local address 127.27.209.254:36299)
0809 06:49:54.163763 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.163767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.164832 (+ 1065us) server_negotiation.cc:407] Connection header received
0809 06:49:54.166891 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.166894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.166953 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.167039 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:54.168629 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.169546 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.171079 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.171729 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.171854 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.175659 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:54.175665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:54.175669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:54.175706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:54.178596 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.179245 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.179250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.179254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.179358 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.179806 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.179811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.179815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.180061 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:54.180192 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.180871 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.181073 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":562,"thread_start_us":461,"threads_started":1}
I20260809 06:49:54.182333 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.163913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:59921)
0809 06:49:54.164656 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.164726 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:54.166602 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:54.167320 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:54.167328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:54.167876 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:49:54.168418 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.168436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.169740 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.169744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.170936 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.170948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.171923 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.171928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.172150 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.172778 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:49:54.172808 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:49:54.175416 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:49:54.178721 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:54.178730 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:54.178747 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:54.179136 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:54.179460 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:49:54.179465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:49:54.179469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:49:54.179704 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:49:54.180193 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:49:54.180200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:49:54.180518 (+ 318us) client_negotiation.cc:769] Sending connection context
0809 06:49:54.180752 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 06:49:54.181240 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":314,"threads_started":1}
I20260809 06:49:54.183521 21036 heartbeater.cc:347] Connected to a master server at 127.27.209.254:36299
I20260809 06:49:54.183843 21036 heartbeater.cc:464] Registering TS with master...
I20260809 06:49:54.184563 21036 heartbeater.cc:511] Master 127.27.209.254:36299 requested a full tablet report, sending...
I20260809 06:49:54.185899 20396 ts_manager.cc:194] Registered new tserver with Master: 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
I20260809 06:49:54.186674 20396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:59921
I20260809 06:49:54.192456 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:49:54.199138 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.193318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.27.209.254:36299)
0809 06:49:54.193474 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.193478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.193915 (+ 437us) server_negotiation.cc:407] Connection header received
0809 06:49:54.194090 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.194094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.194234 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.194355 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:49:54.194824 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.195506 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.197208 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.197714 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.197824 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.198316 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:49:54.198565 (+ 249us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 06:49:54.198889 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34534: token expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260809 06:49:54.199234 21039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34534: token expired
W20260809 06:49:54.199790 20278 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.209.254:36299)
I20260809 06:49:54.211578 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.200207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.27.209.254:36299)
0809 06:49:54.200351 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.200355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.200387 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:49:54.200525 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.200529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.200578 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.200646 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:54.201908 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.202562 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.203890 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.204276 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.204368 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.206902 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:54.206907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:54.206910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:54.206938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:54.209471 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.210065 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.210069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.210071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.210182 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.210457 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.210460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.210462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.210729 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:54.210920 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.211137 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.211280 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:49:54.214686 20396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:49:54.222005 20396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:49:54.255983 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.233856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33422)
0809 06:49:54.245252 (+ 11396us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.245272 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:54.245395 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:54.248450 (+ 3055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:54.248457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:54.248489 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:54.248894 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.248916 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.251043 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.251049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.252996 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.253009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.254850 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.254857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.255152 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.255207 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 06:49:54.255344 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 06:49:54.255519 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11260,"mutex_wait_us":21,"thread_start_us":15496,"threads_started":1}
I20260809 06:49:54.256718 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.234163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.27.209.193:40579)
0809 06:49:54.247753 (+ 13590us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.247762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.247794 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:49:54.247896 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.247904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.248186 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.249119 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:54.249388 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.250862 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.253162 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.254607 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.254781 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.254961 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.256223 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.256397 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13397,"thread_start_us":14283,"threads_started":1}
I20260809 06:49:54.257090 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.233585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40831 (local address 127.0.0.1:45568)
0809 06:49:54.243301 (+ 9716us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.243352 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:54.243525 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:54.250390 (+ 6865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:54.250395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:54.250419 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:54.250882 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.250895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.252827 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.252832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.254327 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.254341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.256301 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.256307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.256464 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.256498 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:49:54.256591 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 06:49:54.256712 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9433,"mutex_wait_us":23,"thread_start_us":11389,"threads_started":1}
I20260809 06:49:54.257130 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.233584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.27.209.194:40831)
0809 06:49:54.249927 (+ 16343us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.249935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.249965 (+ 30us) server_negotiation.cc:407] Connection header received
0809 06:49:54.250042 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.250047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.250212 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.250389 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:54.251102 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.252672 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.254483 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.255825 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.255988 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.256161 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.256662 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.256842 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16199,"thread_start_us":16107,"threads_started":1}
I20260809 06:49:54.258675 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.233982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44013 (local address 127.0.0.1:59934)
0809 06:49:54.249639 (+ 15657us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.249657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:54.249789 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:54.250609 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:54.250614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:54.250641 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:54.251084 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.251095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.252994 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.252997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.255087 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.255100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.257090 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.257096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.258072 (+ 976us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.258105 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:49:54.258212 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 06:49:54.258329 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15527,"mutex_wait_us":38,"thread_start_us":9743,"threads_started":1}
I20260809 06:49:54.258759 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.234572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.27.209.195:44013)
0809 06:49:54.249184 (+ 14612us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.249192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.249903 (+ 711us) server_negotiation.cc:407] Connection header received
0809 06:49:54.249999 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.250006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.250217 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.250396 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:54.251306 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.252861 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.255275 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.256565 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.256744 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.256907 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.258286 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.258445 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14432,"thread_start_us":14283,"threads_started":1}
I20260809 06:49:54.261109 20753 tablet_service.cc:1467] Processing CreateTablet for tablet aec596f35eaf458b89f74973d20b5fed (DEFAULT_TABLE table=test-table [id=23e289bbda60492caa7e6a2b0fa44f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:54.261126 20572 tablet_service.cc:1467] Processing CreateTablet for tablet aec596f35eaf458b89f74973d20b5fed (DEFAULT_TABLE table=test-table [id=23e289bbda60492caa7e6a2b0fa44f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:54.262513 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec596f35eaf458b89f74973d20b5fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:54.262569 20572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec596f35eaf458b89f74973d20b5fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:54.262169 20937 tablet_service.cc:1467] Processing CreateTablet for tablet aec596f35eaf458b89f74973d20b5fed (DEFAULT_TABLE table=test-table [id=23e289bbda60492caa7e6a2b0fa44f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:49:54.263451 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec596f35eaf458b89f74973d20b5fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:49:54.276901 21061 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Bootstrap starting.
I20260809 06:49:54.279793 21062 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Bootstrap starting.
I20260809 06:49:54.280913 21061 tablet_bootstrap.cc:654] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:54.281276 21063 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Bootstrap starting.
I20260809 06:49:54.282195 21061 log.cc:824] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:54.283195 21062 tablet_bootstrap.cc:654] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:54.284271 21063 tablet_bootstrap.cc:654] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Neither blocks nor log segments found. Creating new log.
I20260809 06:49:54.284356 21062 log.cc:824] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:54.285354 21063 log.cc:824] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Log is configured to *not* fsync() on all Append() calls
I20260809 06:49:54.285410 21061 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: No bootstrap required, opened a new log
I20260809 06:49:54.285681 21061 ts_tablet_manager.cc:1397] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:49:54.287523 21062 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: No bootstrap required, opened a new log
I20260809 06:49:54.287906 21062 ts_tablet_manager.cc:1397] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260809 06:49:54.290225 21063 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: No bootstrap required, opened a new log
I20260809 06:49:54.290493 21063 ts_tablet_manager.cc:1397] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260809 06:49:54.293519 21061 raft_consensus.cc:348] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.293869 21061 raft_consensus.cc:374] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:54.293967 21061 raft_consensus.cc:729] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a364b2ba4149378e9c3343802b3fba, State: Initialized, Role: FOLLOWER
I20260809 06:49:54.294521 21061 consensus_queue.cc:260] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.295598 20842 heartbeater.cc:503] Master 127.27.209.254:36299 was elected leader, sending a full tablet report...
I20260809 06:49:54.296275 21061 ts_tablet_manager.cc:1428] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260809 06:49:54.296590 20659 heartbeater.cc:503] Master 127.27.209.254:36299 was elected leader, sending a full tablet report...
I20260809 06:49:54.299815 21062 raft_consensus.cc:348] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.300225 21062 raft_consensus.cc:374] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:54.300339 21062 raft_consensus.cc:729] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dec15678af4232ad692b3c86d092b7, State: Initialized, Role: FOLLOWER
I20260809 06:49:54.300654 21063 raft_consensus.cc:348] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.301065 21063 raft_consensus.cc:374] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:49:54.300963 21062 consensus_queue.cc:260] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.301162 21063 raft_consensus.cc:729] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40088212fd5449848eb22ed6ae9fd6a2, State: Initialized, Role: FOLLOWER
I20260809 06:49:54.301755 21063 consensus_queue.cc:260] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.302129 21036 heartbeater.cc:503] Master 127.27.209.254:36299 was elected leader, sending a full tablet report...
I20260809 06:49:54.302873 21062 ts_tablet_manager.cc:1428] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260809 06:49:54.303434 21063 ts_tablet_manager.cc:1428] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260809 06:49:54.412101 21037 tablet.cc:2369] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:54.462877 21070 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:49:54.463191 21070 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.465046 21070 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
W20260809 06:49:54.470608 20844 tablet.cc:2369] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:54.472218 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.465626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:56515 (local address 127.27.209.194:40831)
0809 06:49:54.465842 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.465847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.466730 (+ 883us) server_negotiation.cc:407] Connection header received
0809 06:49:54.466915 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.466919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.466978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.467051 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:54.467730 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.468444 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.470436 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.471331 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.471467 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.471565 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.471844 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.471966 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:49:54.472535 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.465479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:59151 (local address 127.27.209.195:44013)
0809 06:49:54.465700 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.465707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.466641 (+ 934us) server_negotiation.cc:407] Connection header received
0809 06:49:54.466845 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.466849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.466910 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.466984 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:49:54.467687 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.468415 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.470472 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.471678 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.471820 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.471895 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.472193 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.472319 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 06:49:54.472741 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.465623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40831 (local address 127.27.209.193:56515)
0809 06:49:54.466638 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.466656 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:54.466788 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:54.467082 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:54.467086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:54.467115 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:54.467517 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.467551 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.468569 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.468575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.470270 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.470283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.471526 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.471554 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.471693 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.471716 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:49:54.471798 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 06:49:54.471942 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":32,"thread_start_us":591,"threads_started":1}
I20260809 06:49:54.472786 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.465464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44013 (local address 127.27.209.193:59151)
0809 06:49:54.466527 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.466544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:49:54.466687 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:49:54.467014 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:49:54.467028 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:49:54.467054 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:49:54.467503 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.467514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.468548 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.468553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.470271 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:49:54.470287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.471883 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.471889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:49:54.472049 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.472075 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 06:49:54.472165 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 06:49:54.472289 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":75,"spinlock_wait_cycles":19840,"thread_start_us":581,"threads_started":1}
I20260809 06:49:54.473734 20783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec596f35eaf458b89f74973d20b5fed" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8a364b2ba4149378e9c3343802b3fba" is_pre_election: true
I20260809 06:49:54.473740 20972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec596f35eaf458b89f74973d20b5fed" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dec15678af4232ad692b3c86d092b7" is_pre_election: true
I20260809 06:49:54.474206 20783 raft_consensus.cc:2454] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 0.
I20260809 06:49:54.474200 20972 raft_consensus.cc:2454] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 0.
I20260809 06:49:54.474817 20504 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dec15678af4232ad692b3c86d092b7, 40088212fd5449848eb22ed6ae9fd6a2; no voters:
I20260809 06:49:54.475174 21070 raft_consensus.cc:2781] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:49:54.475338 21070 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:49:54.475442 21070 raft_consensus.cc:3037] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:54.477620 21070 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.478171 21070 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 election: Requested vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
I20260809 06:49:54.478585 20783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec596f35eaf458b89f74973d20b5fed" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8a364b2ba4149378e9c3343802b3fba"
I20260809 06:49:54.478606 20972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec596f35eaf458b89f74973d20b5fed" candidate_uuid: "40088212fd5449848eb22ed6ae9fd6a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dec15678af4232ad692b3c86d092b7"
I20260809 06:49:54.478776 20972 raft_consensus.cc:3037] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:54.478775 20783 raft_consensus.cc:3037] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:49:54.480994 20972 raft_consensus.cc:2454] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 1.
I20260809 06:49:54.481087 20783 raft_consensus.cc:2454] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40088212fd5449848eb22ed6ae9fd6a2 in term 1.
I20260809 06:49:54.481374 20504 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dec15678af4232ad692b3c86d092b7, 40088212fd5449848eb22ed6ae9fd6a2; no voters:
I20260809 06:49:54.481684 21070 raft_consensus.cc:2781] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:49:54.482167 21070 raft_consensus.cc:686] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 LEADER]: Becoming Leader. State: Replica: 40088212fd5449848eb22ed6ae9fd6a2, State: Running, Role: LEADER
I20260809 06:49:54.482641 21070 consensus_queue.cc:237] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:49:54.486768 20395 catalog_manager.cc:5495] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40088212fd5449848eb22ed6ae9fd6a2 (127.27.209.193). New cstate: current_term: 1 leader_uuid: "40088212fd5449848eb22ed6ae9fd6a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } health_report { overall_health: UNKNOWN } } }
W20260809 06:49:54.513801 20660 tablet.cc:2369] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:49:54.551082 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.546045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.27.209.193:40579)
0809 06:49:54.546216 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.546220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.546245 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:49:54.546419 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.546422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.546499 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.546617 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:49:54.546911 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.547705 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.549092 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.549491 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.549607 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.550090 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:49:54.550576 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:49:54.550706 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.550744 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.550809 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 06:49:54.558838 20972 raft_consensus.cc:1261] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Refusing update from remote peer 40088212fd5449848eb22ed6ae9fd6a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:54.558889 20783 raft_consensus.cc:1261] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Refusing update from remote peer 40088212fd5449848eb22ed6ae9fd6a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:49:54.559806 21074 consensus_queue.cc:1035] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:54.560186 21070 consensus_queue.cc:1035] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:49:54.588522 20395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:54.588784 20395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:49:54.592067 20395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Sending DeleteTablet for 3 replicas of tablet aec596f35eaf458b89f74973d20b5fed
I20260809 06:49:54.593204 20572 tablet_service.cc:1514] Processing DeleteTablet for tablet aec596f35eaf458b89f74973d20b5fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:54 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:33422
I20260809 06:49:54.593240 20937 tablet_service.cc:1514] Processing DeleteTablet for tablet aec596f35eaf458b89f74973d20b5fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:54 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:59934
I20260809 06:49:54.593742 20753 tablet_service.cc:1514] Processing DeleteTablet for tablet aec596f35eaf458b89f74973d20b5fed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:49:54 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:45568
I20260809 06:49:54.594614 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 20446
I20260809 06:49:54.606185 21096 tablet_replica.cc:330] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: stopping tablet replica
I20260809 06:49:54.606681 21096 raft_consensus.cc:2229] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:54.607410 21097 tablet_replica.cc:330] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: stopping tablet replica
I20260809 06:49:54.607930 21097 raft_consensus.cc:2229] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:49:54.608552 21097 raft_consensus.cc:2258] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:54.609081 21096 raft_consensus.cc:2258] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:54.610797 21095 tablet_replica.cc:330] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: stopping tablet replica
I20260809 06:49:54.611196 21095 raft_consensus.cc:2229] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:49:54.612042 21095 raft_consensus.cc:2258] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:49:54.612228 21096 ts_tablet_manager.cc:1905] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:54.614326 21097 ts_tablet_manager.cc:1905] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:54.616158 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.595707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.27.209.193:40579)
0809 06:49:54.595882 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.595889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.595908 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:49:54.595962 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.595967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.596048 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.596161 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:54.598153 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.599443 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.601414 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.602077 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.602218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.606744 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:54.606759 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:54.606777 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:54.606817 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:54.611069 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.612158 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.612167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.612171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.612341 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.612830 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.612835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.612838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.613877 (+ 1039us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:54.614311 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.615513 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.615779 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:49:54.618132 20653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:54.619331 21096 ts_tablet_manager.cc:1918] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:54.619552 21096 log.cc:1196] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/aec596f35eaf458b89f74973d20b5fed
I20260809 06:49:54.620146 21096 ts_tablet_manager.cc:1939] T aec596f35eaf458b89f74973d20b5fed P 28dec15678af4232ad692b3c86d092b7: Deleting consensus metadata
I20260809 06:49:54.621261 21097 ts_tablet_manager.cc:1918] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:49:54.621419 21097 log.cc:1196] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/aec596f35eaf458b89f74973d20b5fed
I20260809 06:49:54.621963 21097 ts_tablet_manager.cc:1939] T aec596f35eaf458b89f74973d20b5fed P b8a364b2ba4149378e9c3343802b3fba: Deleting consensus metadata
I20260809 06:49:54.622061 20375 catalog_manager.cc:4841] TS 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): tablet aec596f35eaf458b89f74973d20b5fed (table test-table [id=23e289bbda60492caa7e6a2b0fa44f4c]) successfully deleted
I20260809 06:49:54.623699 20376 catalog_manager.cc:4841] TS b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): tablet aec596f35eaf458b89f74973d20b5fed (table test-table [id=23e289bbda60492caa7e6a2b0fa44f4c]) successfully deleted
I20260809 06:49:54.619046 21095 ts_tablet_manager.cc:1905] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:49:54.716857 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 20446
I20260809 06:49:54.746529 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 20665
W20260809 06:49:54.746461 20378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.209.193:40579 (error 108)
I20260809 06:49:54.748240 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.747853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33450)
0809 06:49:54.748003 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.748096 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 06:49:54.748479 20378 catalog_manager.cc:4584] TS 40088212fd5449848eb22ed6ae9fd6a2 (127.27.209.193:40579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aec596f35eaf458b89f74973d20b5fed: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
I20260809 06:49:54.760421 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.747314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.27.209.194:40831)
0809 06:49:54.747502 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.747508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.747664 (+ 156us) server_negotiation.cc:407] Connection header received
0809 06:49:54.747877 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.747882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.747946 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.748037 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:54.749785 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.750511 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.752003 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.752394 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.752504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.754968 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:54.754979 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:54.754993 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:54.755028 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:54.757790 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.758447 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.758455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.758458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.758588 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.758992 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.758995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.758998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.759375 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:54.759690 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.759972 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.760143 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:49:54.761046 20837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:54.802083 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.801539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33452)
0809 06:49:54.801779 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.801921 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 06:49:54.803221 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.802809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33458)
0809 06:49:54.802978 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.803080 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 06:49:54.856751 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 20665
I20260809 06:49:54.864039 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.863607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33470)
0809 06:49:54.863788 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.863870 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 06:49:54.865476 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.865118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33478)
0809 06:49:54.865287 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.865358 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 06:49:54.880412 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 20847
I20260809 06:49:54.893882 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.881156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.27.209.195:44013)
0809 06:49:54.881326 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 06:49:54.881333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:49:54.881384 (+ 51us) server_negotiation.cc:407] Connection header received
0809 06:49:54.881589 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:49:54.881594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:49:54.881693 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:49:54.881777 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:49:54.883248 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.884001 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.885325 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:49:54.885747 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:49:54.885871 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:49:54.888577 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:49:54.888592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:49:54.888610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:49:54.888653 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:49:54.891105 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.891789 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.891797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.891799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.891948 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:49:54.892326 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:49:54.892330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:49:54.892333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:49:54.892710 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:49:54.892993 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 06:49:54.893328 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 06:49:54.893539 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:49:54.894495 21025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:54.946194 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.945566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33482)
0809 06:49:54.945831 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.945952 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 06:49:54.947422 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:49:54.946948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:40579 (local address 127.0.0.1:33496)
0809 06:49:54.947178 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 06:49:54.947272 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260809 06:49:54.983234 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 20847
I20260809 06:49:55.003564 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 20346
I20260809 06:49:55.004817 20437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:49:55.118172 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 20346
I20260809 06:50:15.148211 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36299
--webserver_interface=127.27.209.254
--webserver_port=44565
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:15.493757 21106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:15.494053 21106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:15.494196 21106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:15.501317 21106 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 06:50:15.501423 21106 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 06:50:15.501490 21106 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 06:50:15.501560 21106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:15.501679 21106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:15.501777 21106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:15.501844 21106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:15.501921 21106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:15.511638 21106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:36299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=44565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:15.512883 21106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:15.514214 21106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:15.526342 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:15.526433 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:15.526369 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:15.526623 21106 server_base.cc:1034] running on GCE node
I20260809 06:50:15.527776 21106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:15.529183 21106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:15.530409 21106 hybrid_clock.cc:648] HybridClock initialized: now 1786258215530347 us; error 31 us; skew 500 ppm
I20260809 06:50:15.530714 21106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:15 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258215, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:15.538380 21106 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:50:15.539842 21106 webserver.cc:460] Webserver started at http://127.27.209.254:44565/ using document root <none> and password file <none>
I20260809 06:50:15.540519 21106 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:15.540613 21106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:15.545953 21106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260809 06:50:15.548804 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:15.550081 21106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:15.550241 21106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "95544c1c3fc342329d592b81f88b91ff"
format_stamp: "Formatted at 2026-08-09 06:49:29 on dist-test-slave-jpj1"
server_key: "d45d8caf8f57366e7e0ca9cbc7c11f9a"
server_key_iv: "cfc0b7a75684ee9ead1d0ed4ed7b6ea0"
server_key_version: "encryptionkey@0"
I20260809 06:50:15.551168 21106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:15.565647 21106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:15.569947 21106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:15.570184 21106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:15.587683 21106 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:36299
I20260809 06:50:15.587720 21202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:36299 every 8 connection(s)
I20260809 06:50:15.589113 21106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:15.595952 21204 sys_catalog.cc:263] Verifying existing consensus state
I20260809 06:50:15.597337 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 21106
I20260809 06:50:15.598227 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:40579
--local_ip_for_outbound_sockets=127.27.209.193
--tserver_master_addrs=127.27.209.254:36299
--webserver_port=45963
--webserver_interface=127.27.209.193
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:50:15.599261 21204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap starting.
I20260809 06:50:15.619513 21204 log.cc:824] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:15.635871 21204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:50:15.636404 21204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Bootstrap complete.
I20260809 06:50:15.645408 21204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:50:15.646298 21204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95544c1c3fc342329d592b81f88b91ff, State: Initialized, Role: FOLLOWER
I20260809 06:50:15.646955 21204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:50:15.647195 21204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:15.647311 21204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:15.647480 21204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 2 FOLLOWER]: Advancing to term 3
I20260809 06:50:15.651321 21204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:50:15.651826 21204 leader_election.cc:302] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95544c1c3fc342329d592b81f88b91ff; no voters:
I20260809 06:50:15.652467 21204 leader_election.cc:288] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 06:50:15.652623 21208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 3 FOLLOWER]: Leader election won for term 3
I20260809 06:50:15.653874 21208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [term 3 LEADER]: Becoming Leader. State: Replica: 95544c1c3fc342329d592b81f88b91ff, State: Running, Role: LEADER
I20260809 06:50:15.654533 21208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } }
I20260809 06:50:15.654728 21204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:15.657148 21214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "95544c1c3fc342329d592b81f88b91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } } }
I20260809 06:50:15.657508 21214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:15.658962 21215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95544c1c3fc342329d592b81f88b91ff. Latest consensus state: current_term: 3 leader_uuid: "95544c1c3fc342329d592b81f88b91ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95544c1c3fc342329d592b81f88b91ff" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 36299 } } }
I20260809 06:50:15.659242 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:15.659329 21215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:15.668200 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:15.669931 21218 catalog_manager.cc:1259] Loaded cluster ID: f222d4b0185247a89188a8b4b06797c7
I20260809 06:50:15.670092 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:15.674211 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:15.676225 21218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Loaded TSK: 0
I20260809 06:50:15.676437 21218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 95544c1c3fc342329d592b81f88b91ff: Loaded TSK: 1
F20260809 06:50:15.677599 21218 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe0cb25a9ad google::LogMessage::Fail() at ??:0
@ 0x7fe0cb25d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe0cb25a4de google::LogMessage::Flush() at ??:0
@ 0x7fe0cb25ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe0cefbe7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe0cefad73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe0e0c891c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe0e0c88783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe0e0d142a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe0e0d140c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe0e0d5e5e9 std::function<>::operator()() at ??:0
@ 0x7fe0e0c8ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe0e0c8ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe0e0d12610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe0e0d12439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe0cdea6345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe0cdea991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe0cdea9799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe0cde84eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe0d115e6db start_thread at ??:0
@ 0x7fe0ca0f461f clone at ??:0
W20260809 06:50:16.011425 21206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:16.011739 21206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:16.011837 21206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:16.019384 21206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:16.019503 21206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:16.019670 21206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:16.029652 21206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=45963
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:16.030959 21206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:16.032318 21206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:16.045806 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:16.045765 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:16.045949 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:16.046952 21206 server_base.cc:1034] running on GCE node
I20260809 06:50:16.047652 21206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:16.048779 21206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:16.049986 21206 hybrid_clock.cc:648] HybridClock initialized: now 1786258216049939 us; error 42 us; skew 500 ppm
I20260809 06:50:16.050318 21206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:16 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258216, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:16.053515 21206 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:50:16.054903 21206 webserver.cc:460] Webserver started at http://127.27.209.193:45963/ using document root <none> and password file <none>
I20260809 06:50:16.055635 21206 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:16.055760 21206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:16.061174 21206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:50:16.064208 21282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:16.065570 21206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:16.065748 21206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "40088212fd5449848eb22ed6ae9fd6a2"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "e671eec09d7fd20ec4d1811e810d1b9c"
server_key_iv: "c0530e04646d5859cda76ca050ff73a2"
server_key_version: "encryptionkey@0"
I20260809 06:50:16.066722 21206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:16.088991 21206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:16.093240 21206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:16.093518 21206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:16.094535 21206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:16.096504 21290 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 06:50:16.102197 21206 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 06:50:16.102305 21206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260809 06:50:16.102407 21206 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 06:50:16.105515 21206 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 06:50:16.105610 21206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20260809 06:50:16.106050 21290 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Bootstrap starting.
I20260809 06:50:16.116034 21290 log.cc:824] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:16.128372 21290 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 06:50:16.129307 21290 tablet_bootstrap.cc:492] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Bootstrap complete.
I20260809 06:50:16.129967 21290 ts_tablet_manager.cc:1397] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20260809 06:50:16.142053 21290 raft_consensus.cc:348] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:16.143492 21206 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:40579
I20260809 06:50:16.143518 21439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:40579 every 8 connection(s)
I20260809 06:50:16.143610 21290 raft_consensus.cc:729] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40088212fd5449848eb22ed6ae9fd6a2, State: Initialized, Role: FOLLOWER
I20260809 06:50:16.144407 21290 consensus_queue.cc:260] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:16.145089 21206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:16.146292 21290 ts_tablet_manager.cc:1428] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260809 06:50:16.147413 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 21206
I20260809 06:50:16.148351 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:40831
--local_ip_for_outbound_sockets=127.27.209.194
--tserver_master_addrs=127.27.209.254:36299
--webserver_port=33171
--webserver_interface=127.27.209.194
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:50:16.150233 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:16.148364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:55659)
0809 06:50:16.149184 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:16.149412 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":372,"threads_started":1}
W20260809 06:50:16.150887 21441 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:36299 (0 consecutive failures): Network error: Failed to ping master at 127.27.209.254:36299: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
I20260809 06:50:16.151726 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:16.151385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:35859)
0809 06:50:16.151553 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:16.151627 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 06:50:16.152581 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:16.152242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:36071)
0809 06:50:16.152397 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:16.152467 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260809 06:50:16.152830 21441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 06:50:16.482065 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:16.482347 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:16.482472 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:16.489615 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:16.489734 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:16.489879 21447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:16.499357 21447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:40831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=33171
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:16.500600 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:16.501876 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:16.514534 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:16.514562 21458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:16.514976 21457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:16.515388 21447 server_base.cc:1034] running on GCE node
I20260809 06:50:16.516065 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:16.516916 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:16.518137 21447 hybrid_clock.cc:648] HybridClock initialized: now 1786258216518070 us; error 42 us; skew 500 ppm
I20260809 06:50:16.518440 21447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:16 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258216, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:16.521459 21447 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:50:16.522881 21447 webserver.cc:460] Webserver started at http://127.27.209.194:33171/ using document root <none> and password file <none>
I20260809 06:50:16.523518 21447 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:16.523699 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:16.529217 21447 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 06:50:16.532039 21471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:16.533354 21447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:50:16.533517 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "b8a364b2ba4149378e9c3343802b3fba"
format_stamp: "Formatted at 2026-08-09 06:49:30 on dist-test-slave-jpj1"
server_key: "149628ea9438162c73aa990d853b5046"
server_key_iv: "b64262517961511fd8ca2e2d7d19d80f"
server_key_version: "encryptionkey@0"
I20260809 06:50:16.534479 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:16.557039 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:16.561074 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:16.561340 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:16.562330 21447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:16.563823 21447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:16.563923 21447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:16.564034 21447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:16.564105 21447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:16.595433 21447 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:40831
I20260809 06:50:16.595502 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:40831 every 8 connection(s)
I20260809 06:50:16.596906 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:16.601868 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:16.600023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:37355)
0809 06:50:16.600746 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:16.601001 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":269,"threads_started":1}
W20260809 06:50:16.602484 21630 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:36299 (0 consecutive failures): Network error: Failed to ping master at 127.27.209.254:36299: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
I20260809 06:50:16.603318 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:16.602914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:38773)
0809 06:50:16.603118 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:16.603208 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260809 06:50:16.604149 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:16.603838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:40535)
0809 06:50:16.603952 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:16.604025 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 06:50:16.604358 21630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:50:16.606280 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 21447
I20260809 06:50:16.606892 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:44013
--local_ip_for_outbound_sockets=127.27.209.195
--tserver_master_addrs=127.27.209.254:36299
--webserver_port=45043
--webserver_interface=127.27.209.195
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:16.942807 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:16.943081 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:16.943188 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:16.950266 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:16.950372 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:16.950491 21634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:16.960031 21634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=45043
--tserver_master_addrs=127.27.209.254:36299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:16.961288 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:16.962570 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:16.975410 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:16.975474 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:16.975382 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:16.976594 21634 server_base.cc:1034] running on GCE node
I20260809 06:50:16.977174 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:16.978096 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:16.979308 21634 hybrid_clock.cc:648] HybridClock initialized: now 1786258216979252 us; error 48 us; skew 500 ppm
I20260809 06:50:16.979668 21634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:16 dist-test-slave-jpj1 krb5kdc[19537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258216, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:16.983255 21634 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:50:16.985767 21634 webserver.cc:460] Webserver started at http://127.27.209.195:45043/ using document root <none> and password file <none>
I20260809 06:50:16.986459 21634 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:16.986569 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:16.991941 21634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:16.994623 21659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:16.995975 21634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:16.996131 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "28dec15678af4232ad692b3c86d092b7"
format_stamp: "Formatted at 2026-08-09 06:49:31 on dist-test-slave-jpj1"
server_key: "ca7e920fba62cdfe1874eabf61c8c612"
server_key_iv: "b3391cb35de660bb6c786647323611c2"
server_key_version: "encryptionkey@0"
I20260809 06:50:16.997063 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:17.032541 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:17.037086 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:17.037354 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:17.038458 21634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:17.039958 21634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:17.040057 21634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:17.040169 21634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:17.040237 21634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:17.071985 21634 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:44013
I20260809 06:50:17.072043 21829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:44013 every 8 connection(s)
I20260809 06:50:17.073426 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:17.075263 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 21634
I20260809 06:50:17.078707 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.076689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:47099)
0809 06:50:17.077526 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.077817 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":320,"threads_started":1}
W20260809 06:50:17.079329 21830 heartbeater.cc:644] Failed to heartbeat to 127.27.209.254:36299 (0 consecutive failures): Network error: Failed to ping master at 127.27.209.254:36299: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
I20260809 06:50:17.080190 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.079872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:41587)
0809 06:50:17.080011 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.080082 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 06:50:17.080977 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.080683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:53609)
0809 06:50:17.080799 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.080873 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 06:50:17.081187 21830 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 06:50:17.154186 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.153329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:44923)
0809 06:50:17.153806 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.153922 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":199,"threads_started":1}
I20260809 06:50:17.605604 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.604835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:37631)
0809 06:50:17.605286 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.605375 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260809 06:50:17.842976 21838 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:17.843303 21838 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:17.844965 21838 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
Aug 09 06:50:17 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258216, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
Aug 09 06:50:17 dist-test-slave-jpj1 krb5kdc[19537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258216, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:50:17.884070 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.845342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:40831 (local address 127.27.209.193:38771)
0809 06:50:17.858569 (+ 13227us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.858625 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:17.859315 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:17.864583 (+ 5268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:17.864588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:17.865705 (+ 1117us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:17.866921 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:17.866948 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.868960 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.868965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:17.871166 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:17.871186 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.873179 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.873186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:17.873368 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:17.873898 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:17.873916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:17.876431 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:17.880431 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:17.880439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:17.880453 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:17.880919 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:17.881355 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:17.881360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:17.881364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:17.881553 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:17.882368 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:17.882376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:17.882765 (+ 389us) client_negotiation.cc:769] Sending connection context
0809 06:50:17.883060 (+ 295us) client_negotiation.cc:240] Negotiation successful
0809 06:50:17.883598 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13073,"mutex_wait_us":91,"spinlock_wait_cycles":10240,"thread_start_us":15055,"threads_started":1}
I20260809 06:50:17.884136 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.845530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:38771 (local address 127.27.209.194:40831)
0809 06:50:17.857807 (+ 12277us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:17.857828 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:17.859826 (+ 1998us) server_negotiation.cc:407] Connection header received
0809 06:50:17.863142 (+ 3316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:17.863152 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:17.863603 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:17.863959 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:17.867161 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.868659 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.871324 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.872387 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.872573 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:17.876738 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:17.876750 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:17.876770 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:17.876824 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:17.880261 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:17.881060 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:17.881069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:17.881077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:17.881234 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:17.881681 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:17.881686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:17.881690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:17.882221 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:17.882398 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:17.883329 (+ 931us) server_negotiation.cc:299] Negotiation successful
0809 06:50:17.883775 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12034,"thread_start_us":12320,"threads_started":1}
I20260809 06:50:17.884807 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.845374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:44013 (local address 127.27.209.193:43991)
0809 06:50:17.860227 (+ 14853us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:17.860245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:17.860380 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:17.864526 (+ 4146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:17.864538 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:17.865706 (+ 1168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:17.866924 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:17.866950 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.869748 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.869754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:17.871293 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:17.871309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.872556 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.872561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:17.872739 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:17.873738 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:17.873770 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:17.876766 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:17.881279 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:17.881285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:17.881301 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:17.881883 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:17.882508 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:17.882513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:17.882516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:17.882703 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:17.883833 (+ 1130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:17.883838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:17.883992 (+ 154us) client_negotiation.cc:769] Sending connection context
0809 06:50:17.884156 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 06:50:17.884419 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14725,"mutex_wait_us":134,"spinlock_wait_cycles":32384,"thread_start_us":14032,"threads_started":1}
I20260809 06:50:17.886173 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:17.845657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:43991 (local address 127.27.209.195:44013)
0809 06:50:17.860234 (+ 14577us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:17.860250 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:17.860516 (+ 266us) server_negotiation.cc:407] Connection header received
0809 06:50:17.863051 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:17.863063 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:17.863603 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:17.863958 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:17.867192 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.869522 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.871460 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:17.872387 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:17.872589 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:17.877079 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:17.877090 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:17.877107 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:17.877152 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:17.881107 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:17.882077 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:17.882086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:17.882095 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:17.882349 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:17.882841 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:17.882847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:17.882850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:17.883690 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:17.885086 (+ 1396us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:17.885366 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 06:50:17.885767 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14395,"thread_start_us":14276,"threads_started":1}
W20260809 06:50:17.889011 21284 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
W20260809 06:50:17.889637 21283 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
I20260809 06:50:17.889770 21283 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2; no voters: 28dec15678af4232ad692b3c86d092b7, b8a364b2ba4149378e9c3343802b3fba
I20260809 06:50:17.890223 21838 raft_consensus.cc:2726] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:50:18.082563 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:18.081679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:47019)
0809 06:50:18.082188 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:18.082334 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":225,"threads_started":1}
I20260809 06:50:18.155417 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:18.155020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:48081)
0809 06:50:18.155172 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:18.155264 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 06:50:18.607062 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:18.606401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:59747)
0809 06:50:18.606831 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:18.606912 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260809 06:50:19.084343 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:19.083591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:44841)
0809 06:50:19.084079 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:19.084175 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":253,"threads_started":1}
I20260809 06:50:19.157023 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:19.156293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:32953)
0809 06:50:19.156745 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:19.156841 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260809 06:50:19.608636 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:19.607910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:55819)
0809 06:50:19.608364 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:19.608448 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260809 06:50:20.086027 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:20.085274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:50973)
0809 06:50:20.085771 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:20.085861 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20260809 06:50:20.120483 21854 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:20.120673 21854 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:20.121435 21854 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
W20260809 06:50:20.122498 21283 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
W20260809 06:50:20.122655 21284 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
I20260809 06:50:20.122805 21284 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2; no voters: 28dec15678af4232ad692b3c86d092b7, b8a364b2ba4149378e9c3343802b3fba
I20260809 06:50:20.123049 21854 raft_consensus.cc:2726] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:50:20.158723 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:20.157920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:45297)
0809 06:50:20.158414 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:20.158528 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260809 06:50:20.610158 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:20.609455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:42495)
0809 06:50:20.609889 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:20.609972 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260809 06:50:21.087867 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:21.086960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:40117)
0809 06:50:21.087479 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:21.087625 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":222,"threads_started":1}
I20260809 06:50:21.160537 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:21.159734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:35561)
0809 06:50:21.160241 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:21.160341 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":228,"threads_started":1}
I20260809 06:50:21.611752 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:21.611027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:52521)
0809 06:50:21.611451 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:21.611557 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":227,"threads_started":1}
I20260809 06:50:21.735693 21862 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:21.735886 21862 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:21.736475 21862 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
W20260809 06:50:21.737478 21284 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
W20260809 06:50:21.737636 21283 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
I20260809 06:50:21.737780 21283 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2; no voters: 28dec15678af4232ad692b3c86d092b7, b8a364b2ba4149378e9c3343802b3fba
I20260809 06:50:21.738013 21862 raft_consensus.cc:2726] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:50:22.089706 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:22.088919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:47921)
0809 06:50:22.089425 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:22.089525 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":243,"threads_started":1}
I20260809 06:50:22.162240 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:22.161448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:49983)
0809 06:50:22.161959 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:22.162043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":290,"threads_started":1}
I20260809 06:50:22.613408 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:22.612606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:57231)
0809 06:50:22.613050 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:22.613129 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":229,"threads_started":1}
I20260809 06:50:23.091439 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:23.090660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:53561)
0809 06:50:23.091149 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:23.091238 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260809 06:50:23.163971 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:23.163174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:40139)
0809 06:50:23.163665 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:23.163758 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":241,"threads_started":1}
I20260809 06:50:23.615101 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:23.614268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:40935)
0809 06:50:23.614819 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:23.614913 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":237,"threads_started":1}
I20260809 06:50:24.093226 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:24.092437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:38279)
0809 06:50:24.092953 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:24.093036 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":241,"threads_started":1}
I20260809 06:50:24.165565 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:24.164824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:46765)
0809 06:50:24.165304 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:24.165404 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260809 06:50:24.616817 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:24.616005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:37991)
0809 06:50:24.616469 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:24.616585 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
I20260809 06:50:25.094841 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:25.094132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:60497)
0809 06:50:25.094567 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:25.094651 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260809 06:50:25.167132 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:25.166429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:55879)
0809 06:50:25.166878 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:25.166957 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":225,"threads_started":1}
I20260809 06:50:25.526031 21878 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:25.526317 21878 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:25.526986 21878 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
W20260809 06:50:25.527978 21284 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
W20260809 06:50:25.528165 21283 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
I20260809 06:50:25.528282 21283 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2; no voters: 28dec15678af4232ad692b3c86d092b7, b8a364b2ba4149378e9c3343802b3fba
I20260809 06:50:25.528496 21878 raft_consensus.cc:2726] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:50:25.618430 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:25.617703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:59157)
0809 06:50:25.618173 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:25.618273 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":212,"threads_started":1}
I20260809 06:50:26.096547 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:26.095780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:46149)
0809 06:50:26.096302 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:26.096389 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":236,"threads_started":1}
I20260809 06:50:26.168795 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:26.168099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:47997)
0809 06:50:26.168541 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:26.168634 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260809 06:50:26.620035 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:26.619289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:34633)
0809 06:50:26.619767 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:26.619850 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260809 06:50:27.098241 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:27.097472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:52645)
0809 06:50:27.097964 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:27.098052 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":270,"threads_started":1}
I20260809 06:50:27.170621 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:27.169746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:60431)
0809 06:50:27.170269 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:27.170382 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":243,"threads_started":1}
I20260809 06:50:27.621656 21886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:27.620915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:59515)
0809 06:50:27.621379 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:27.621462 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":221,"threads_started":1}
I20260809 06:50:27.814051 21887 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:27.814256 21887 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:27.814939 21887 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
W20260809 06:50:27.815878 21283 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
W20260809 06:50:27.816038 21284 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
I20260809 06:50:27.816180 21284 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2; no voters: 28dec15678af4232ad692b3c86d092b7, b8a364b2ba4149378e9c3343802b3fba
I20260809 06:50:27.816421 21887 raft_consensus.cc:2726] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:50:28.099949 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:28.099171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:45313)
0809 06:50:28.099680 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:28.099781 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":244,"threads_started":1}
I20260809 06:50:28.172469 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:28.171593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:43805)
0809 06:50:28.172094 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:28.172221 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":253,"threads_started":1}
I20260809 06:50:28.623430 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:28.622619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:33691)
0809 06:50:28.623145 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:28.623235 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":274,"threads_started":1}
I20260809 06:50:29.101696 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:29.100961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:51755)
0809 06:50:29.101404 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:29.101491 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260809 06:50:29.174077 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:29.173315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:47397)
0809 06:50:29.173797 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:29.173885 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260809 06:50:29.625005 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:29.624312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:48983)
0809 06:50:29.624747 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:29.624840 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260809 06:50:30.103338 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:30.102608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:43209)
0809 06:50:30.103069 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:30.103169 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260809 06:50:30.175772 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:30.174974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:37935)
0809 06:50:30.175442 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:30.175574 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260809 06:50:30.626680 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:30.625890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:53299)
0809 06:50:30.626385 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:30.626472 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260809 06:50:31.104967 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:31.104191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:47911)
0809 06:50:31.104665 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:31.104753 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
I20260809 06:50:31.177791 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:31.177030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.193:32793)
0809 06:50:31.177513 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:31.177609 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":232,"threads_started":1}
I20260809 06:50:31.249174 21904 raft_consensus.cc:482] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:31.249377 21904 raft_consensus.cc:504] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8a364b2ba4149378e9c3343802b3fba" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 40831 } } peers { permanent_uuid: "40088212fd5449848eb22ed6ae9fd6a2" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 40579 } } peers { permanent_uuid: "28dec15678af4232ad692b3c86d092b7" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 44013 } }
I20260809 06:50:31.250082 21904 leader_election.cc:288] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831), 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013)
W20260809 06:50:31.251161 21283 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28dec15678af4232ad692b3c86d092b7 (127.27.209.195:44013): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
W20260809 06:50:31.251346 21284 leader_election.cc:341] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8a364b2ba4149378e9c3343802b3fba (127.27.209.194:40831): Not found: Tablet not found: aec596f35eaf458b89f74973d20b5fed
I20260809 06:50:31.251515 21284 leader_election.cc:302] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40088212fd5449848eb22ed6ae9fd6a2; no voters: 28dec15678af4232ad692b3c86d092b7, b8a364b2ba4149378e9c3343802b3fba
I20260809 06:50:31.251794 21904 raft_consensus.cc:2726] T aec596f35eaf458b89f74973d20b5fed P 40088212fd5449848eb22ed6ae9fd6a2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 06:50:31.628573 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:31.627786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.194:51745)
0809 06:50:31.628282 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:31.628399 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":225,"threads_started":1}
I20260809 06:50:32.078516 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 21206
I20260809 06:50:32.094204 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:32.079836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.27.209.193:40579)
0809 06:50:32.080350 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:32.080358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:32.080385 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:50:32.080448 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:32.080456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:32.080688 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:32.080841 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:32.082387 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:32.083380 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:32.084749 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:32.085382 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:32.085502 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:32.088224 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:32.088237 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:32.088253 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:32.088292 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:32.091169 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:32.091952 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:32.091959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:32.091961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:32.092103 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:32.092527 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:32.092530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:32.092533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:32.092916 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:32.093223 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:32.093619 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 06:50:32.093894 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260809 06:50:32.095623 21435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:32.106904 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:32.106099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:36299 (local address 127.27.209.195:34777)
0809 06:50:32.106634 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:32.106734 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.254:36299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":249,"threads_started":1}
I20260809 06:50:32.185913 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 21206
I20260809 06:50:32.213725 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 21447
I20260809 06:50:32.227170 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:32.214476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.27.209.194:40831)
0809 06:50:32.215013 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:32.215018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:32.215035 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:50:32.215093 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:32.215097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:32.215166 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:32.215255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:32.216631 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:32.217651 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:32.218944 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:32.219402 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:32.219500 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:32.222230 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:32.222235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:32.222238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:32.222272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:32.224859 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:32.225543 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:32.225548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:32.225550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:32.225703 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:32.226003 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:32.226006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:32.226008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:32.226297 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:32.226518 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:32.226701 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 06:50:32.226870 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":298,"threads_started":1}
I20260809 06:50:32.228022 21621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:32.311520 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 21447
I20260809 06:50:32.336534 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 21634
I20260809 06:50:32.350201 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:32.337427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.27.209.195:44013)
0809 06:50:32.338039 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:32.338044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:32.338062 (+ 18us) server_negotiation.cc:407] Connection header received
0809 06:50:32.338121 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:32.338125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:32.338198 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:32.338303 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:32.339751 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:32.340569 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:32.341834 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:32.342267 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:32.342366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:32.345058 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:32.345063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:32.345067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:32.345111 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:32.347819 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:32.348482 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:32.348487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:32.348489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:32.348643 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:32.348984 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:32.348988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:32.348991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:32.349293 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:32.349504 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:32.349724 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 06:50:32.349898 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":342,"threads_started":1}
I20260809 06:50:32.350943 21823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:32.434907 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 21634
2026-08-09T06:50:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64563 ms)
[----------] 30 tests from SecurityITest (296472 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:50:32 dist-test-slave-jpj1 krb5kdc[21929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:50:32 dist-test-slave-jpj1 krb5kdc[21929](info): set up 2 sockets
Aug 09 06:50:32 dist-test-slave-jpj1 krb5kdc[21929](info): commencing operation
krb5kdc: starting...
W20260809 06:50:33.669951 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:50:33 dist-test-slave-jpj1 krb5kdc[21929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:50:33.945256 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38729
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:34.284020 21951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:34.284319 21951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:34.284457 21951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:34.292023 21951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:34.292141 21951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:34.292223 21951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:34.292308 21951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:34.292368 21951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:34.302350 21951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:34.303695 21951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:34.305028 21951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:34.316859 21961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:34.316947 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:34.316897 21964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:34.317260 21951 server_base.cc:1034] running on GCE node
I20260809 06:50:34.318240 21951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:34.319594 21951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:34.320851 21951 hybrid_clock.cc:648] HybridClock initialized: now 1786258234320790 us; error 36 us; skew 500 ppm
I20260809 06:50:34.321223 21951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:34 dist-test-slave-jpj1 krb5kdc[21929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258234, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:34.328260 21951 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:50:34.329737 21951 webserver.cc:460] Webserver started at http://127.27.209.254:33613/ using document root <none> and password file <none>
I20260809 06:50:34.330428 21951 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:34.330549 21951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:34.330850 21951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:34.333902 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "921fe938412343f69a7851c0e2540661"
format_stamp: "Formatted at 2026-08-09 06:50:34 on dist-test-slave-jpj1"
server_key: "d334b5ad49254940111bd264f4239ec9"
server_key_iv: "45d4d13bae6d77d6f00720a06d64f64a"
server_key_version: "encryptionkey@0"
I20260809 06:50:34.334681 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "921fe938412343f69a7851c0e2540661"
format_stamp: "Formatted at 2026-08-09 06:50:34 on dist-test-slave-jpj1"
server_key: "d334b5ad49254940111bd264f4239ec9"
server_key_iv: "45d4d13bae6d77d6f00720a06d64f64a"
server_key_version: "encryptionkey@0"
I20260809 06:50:34.340175 21951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:50:34.343761 21980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:34.345168 21951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:50:34.345340 21951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "921fe938412343f69a7851c0e2540661"
format_stamp: "Formatted at 2026-08-09 06:50:34 on dist-test-slave-jpj1"
server_key: "d334b5ad49254940111bd264f4239ec9"
server_key_iv: "45d4d13bae6d77d6f00720a06d64f64a"
server_key_version: "encryptionkey@0"
I20260809 06:50:34.345530 21951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:34.407442 21951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:34.411924 21951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:34.412215 21951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:34.430621 21951 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:38729
I20260809 06:50:34.430644 22051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:38729 every 8 connection(s)
I20260809 06:50:34.432214 21951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:34.436170 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:34.442252 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 21951
I20260809 06:50:34.442440 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:34.443121 28487 external_mini_cluster.cc:1428] Setting key f91e9f87630f636a3b31f84ede09b4e3
I20260809 06:50:34.446766 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661: Bootstrap starting.
I20260809 06:50:34.450345 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:34.451517 22052 log.cc:824] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:34.454706 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661: No bootstrap required, opened a new log
Aug 09 06:50:34 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:34.463766 22052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "921fe938412343f69a7851c0e2540661" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38729 } }
I20260809 06:50:34.464252 22052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:34.464367 22052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921fe938412343f69a7851c0e2540661, State: Initialized, Role: FOLLOWER
I20260809 06:50:34.465060 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "921fe938412343f69a7851c0e2540661" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38729 } }
I20260809 06:50:34.465288 22052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:34.465432 22052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:34.465619 22052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:34.467634 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:34.445260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.27.209.254:38729)
0809 06:50:34.446106 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:34.446118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:34.446161 (+ 43us) server_negotiation.cc:407] Connection header received
0809 06:50:34.447965 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:34.447989 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:34.448379 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:34.448748 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:34.450787 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:34.452421 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:34.454223 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:34.454925 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:34.455062 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:34.460075 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:34.460083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:34.460102 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:34.460150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:34.463191 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:34.464088 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:34.464095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:34.464101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:34.464294 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:34.464753 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:34.464759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:34.464762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:34.465396 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:34.465730 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:34.466148 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 06:50:34.466533 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":387,"threads_started":1}
I20260809 06:50:34.468340 22052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "921fe938412343f69a7851c0e2540661" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38729 } }
I20260809 06:50:34.468820 22052 leader_election.cc:302] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 921fe938412343f69a7851c0e2540661; no voters:
I20260809 06:50:34.469453 22052 leader_election.cc:288] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:34.469602 22060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:34.471769 22052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:34.471807 22060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [term 1 LEADER]: Becoming Leader. State: Replica: 921fe938412343f69a7851c0e2540661, State: Running, Role: LEADER
I20260809 06:50:34.472461 22060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "921fe938412343f69a7851c0e2540661" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38729 } }
I20260809 06:50:34.475561 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 921fe938412343f69a7851c0e2540661. Latest consensus state: current_term: 1 leader_uuid: "921fe938412343f69a7851c0e2540661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "921fe938412343f69a7851c0e2540661" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38729 } } }
I20260809 06:50:34.476083 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:34.476008 22061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "921fe938412343f69a7851c0e2540661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "921fe938412343f69a7851c0e2540661" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 38729 } } }
I20260809 06:50:34.476284 22061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:34.477552 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:34.482975 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:34.491648 22076 catalog_manager.cc:1347] Generated new cluster ID: c0c7d7ddb24745479102fd66df3669c7
I20260809 06:50:34.491767 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:34.502158 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:34.503427 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:34.509449 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661: Generated new TSK 0
I20260809 06:50:34.510239 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:50:34.581173 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38729
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:35.008852 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:35.009179 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:35.009320 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:35.016633 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:35.016772 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:35.016925 22091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:35.026953 22091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:35.028380 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:35.029763 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:35.043520 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:35.043617 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:35.043589 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:35.044461 22091 server_base.cc:1034] running on GCE node
I20260809 06:50:35.045086 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:35.046052 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:35.047266 22091 hybrid_clock.cc:648] HybridClock initialized: now 1786258235047208 us; error 42 us; skew 500 ppm
I20260809 06:50:35.047649 22091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:35 dist-test-slave-jpj1 krb5kdc[21929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:35.051021 22091 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:50:35.052532 22091 webserver.cc:460] Webserver started at http://127.27.209.193:37887/ using document root <none> and password file <none>
I20260809 06:50:35.053238 22091 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:35.053355 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:35.053663 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:35.056648 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "14e1e28bce6443f588dc35a96297d3b4"
format_stamp: "Formatted at 2026-08-09 06:50:35 on dist-test-slave-jpj1"
server_key: "57859472f640fb3e96a4e9fc15bb8275"
server_key_iv: "ea0688ce68c4290c34281e8b74fdd65f"
server_key_version: "encryptionkey@0"
I20260809 06:50:35.057415 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "14e1e28bce6443f588dc35a96297d3b4"
format_stamp: "Formatted at 2026-08-09 06:50:35 on dist-test-slave-jpj1"
server_key: "57859472f640fb3e96a4e9fc15bb8275"
server_key_iv: "ea0688ce68c4290c34281e8b74fdd65f"
server_key_version: "encryptionkey@0"
I20260809 06:50:35.062564 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:50:35.066020 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:35.067339 22091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:50:35.067502 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "14e1e28bce6443f588dc35a96297d3b4"
format_stamp: "Formatted at 2026-08-09 06:50:35 on dist-test-slave-jpj1"
server_key: "57859472f640fb3e96a4e9fc15bb8275"
server_key_iv: "ea0688ce68c4290c34281e8b74fdd65f"
server_key_version: "encryptionkey@0"
I20260809 06:50:35.067725 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:35.102487 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:35.106659 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:35.106935 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:35.107949 22091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:35.109328 22091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:35.109406 22091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:35.109539 22091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:35.109613 22091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:35.141050 22091 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:35641
I20260809 06:50:35.141119 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:35641 every 8 connection(s)
I20260809 06:50:35.142319 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:35.147862 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 22091
I20260809 06:50:35.148061 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:50:35.148892 28487 external_mini_cluster.cc:1428] Setting key 7dafbe58dc6ad114bc8ec3d63f91a85f
Aug 09 06:50:35 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:35.162433 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:35.144910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:50143 (local address 127.27.209.254:38729)
0809 06:50:35.145408 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:35.145414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:35.146397 (+ 983us) server_negotiation.cc:407] Connection header received
0809 06:50:35.148419 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:35.148426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:35.148519 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:35.148646 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:35.150463 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.151190 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.152744 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.153146 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.153258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:35.156778 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:35.156783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:35.156786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:35.156821 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:35.159289 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:35.160058 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:35.160062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:35.160064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:35.160205 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:35.160641 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:35.160644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:35.160646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:35.160929 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:35.161065 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:35.161800 (+ 735us) server_negotiation.cc:299] Negotiation successful
0809 06:50:35.162047 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260809 06:50:35.163403 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:35.145523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38729 (local address 127.27.209.193:50143)
0809 06:50:35.146223 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:35.146270 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:35.148094 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:35.148916 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:35.148926 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:35.149551 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:35.150265 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:35.150286 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.151331 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.151335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:35.152608 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:35.152621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.153369 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.153378 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:35.153582 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:35.154269 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:35.154298 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:35.156549 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:35.159429 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:35.159438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:35.159456 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:35.159930 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:35.160314 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:35.160318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:35.160323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:35.160522 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:35.161058 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:35.161067 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:35.161415 (+ 348us) client_negotiation.cc:769] Sending connection context
0809 06:50:35.161689 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 06:50:35.162226 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":269,"threads_started":1}
I20260809 06:50:35.165057 22292 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38729
I20260809 06:50:35.165395 22292 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:35.166136 22292 heartbeater.cc:511] Master 127.27.209.254:38729 requested a full tablet report, sending...
I20260809 06:50:35.168444 22003 ts_manager.cc:194] Registered new tserver with Master: 14e1e28bce6443f588dc35a96297d3b4 (127.27.209.193:35641)
I20260809 06:50:35.170499 22003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:50143
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:50:35.213979 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38729
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:35.545017 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:35.545300 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:35.545431 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:35.552485 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:35.552609 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:35.552763 22300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:35.562114 22300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:35.563333 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:35.564669 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:35.577592 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:35.577557 22310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:35.577652 22308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:35.578974 22300 server_base.cc:1034] running on GCE node
I20260809 06:50:35.579596 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:35.580461 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:35.581745 22300 hybrid_clock.cc:648] HybridClock initialized: now 1786258235581673 us; error 58 us; skew 500 ppm
I20260809 06:50:35.582063 22300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:35 dist-test-slave-jpj1 krb5kdc[21929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:35.585165 22300 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:50:35.586555 22300 webserver.cc:460] Webserver started at http://127.27.209.194:37499/ using document root <none> and password file <none>
I20260809 06:50:35.587237 22300 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:35.587359 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:35.587760 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:35.590646 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "4a3d6122a5994af69860e2ea1a1432e5"
format_stamp: "Formatted at 2026-08-09 06:50:35 on dist-test-slave-jpj1"
server_key: "5b9f0bcf971644fc66bb0d891fbdb83d"
server_key_iv: "650581aad812808f0403e0227d401e1b"
server_key_version: "encryptionkey@0"
I20260809 06:50:35.591394 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "4a3d6122a5994af69860e2ea1a1432e5"
format_stamp: "Formatted at 2026-08-09 06:50:35 on dist-test-slave-jpj1"
server_key: "5b9f0bcf971644fc66bb0d891fbdb83d"
server_key_iv: "650581aad812808f0403e0227d401e1b"
server_key_version: "encryptionkey@0"
I20260809 06:50:35.596421 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260809 06:50:35.599804 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:35.601107 22300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:50:35.601259 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "4a3d6122a5994af69860e2ea1a1432e5"
format_stamp: "Formatted at 2026-08-09 06:50:35 on dist-test-slave-jpj1"
server_key: "5b9f0bcf971644fc66bb0d891fbdb83d"
server_key_iv: "650581aad812808f0403e0227d401e1b"
server_key_version: "encryptionkey@0"
I20260809 06:50:35.601428 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:35.615250 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:35.618960 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:35.619233 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:35.620208 22300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:35.621506 22300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:35.621598 22300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:35.621726 22300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:35.621788 22300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:35.653442 22300 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:38979
I20260809 06:50:35.653488 22476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:38979 every 8 connection(s)
I20260809 06:50:35.654742 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:35.660425 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 22300
I20260809 06:50:35.660589 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:50:35.661159 28487 external_mini_cluster.cc:1428] Setting key 71b521e5bd3c6ed64c9127a335979217
Aug 09 06:50:35 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:35.675643 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:35.657361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:55755 (local address 127.27.209.254:38729)
0809 06:50:35.657578 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:35.657584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:35.658953 (+ 1369us) server_negotiation.cc:407] Connection header received
0809 06:50:35.660850 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:35.660856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:35.660947 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:35.661065 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:35.662671 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.663595 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.665315 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.665908 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.666026 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:35.669350 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:35.669356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:35.669360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:35.669394 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:35.672433 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:35.673163 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:35.673169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:35.673172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:35.673340 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:35.673757 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:35.673761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:35.673764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:35.674109 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:35.674261 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:35.675022 (+ 761us) server_negotiation.cc:299] Negotiation successful
0809 06:50:35.675258 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:50:35.676789 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:35.658005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38729 (local address 127.27.209.194:55755)
0809 06:50:35.658776 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:35.658826 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:35.660546 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:35.661247 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:35.661260 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:35.661891 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:35.662502 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:35.662520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.663765 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.663769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:35.665153 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:35.665167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:35.666105 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:35.666110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:35.666312 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:35.666961 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:35.666984 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:35.669136 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:35.672562 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:35.672571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:35.672588 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:35.673045 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:35.673436 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:35.673439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:35.673441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:35.673631 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:35.674237 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:35.674244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:35.674608 (+ 364us) client_negotiation.cc:769] Sending connection context
0809 06:50:35.674892 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 06:50:35.675369 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":283,"threads_started":1}
I20260809 06:50:35.678269 22480 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38729
I20260809 06:50:35.678584 22480 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:35.679385 22480 heartbeater.cc:511] Master 127.27.209.254:38729 requested a full tablet report, sending...
I20260809 06:50:35.680948 22003 ts_manager.cc:194] Registered new tserver with Master: 4a3d6122a5994af69860e2ea1a1432e5 (127.27.209.194:38979)
I20260809 06:50:35.681769 22003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:55755
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:50:35.724670 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38729
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:36.067306 22485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:36.067672 22485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:36.067834 22485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:36.075362 22485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:36.075469 22485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:36.075661 22485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:36.085549 22485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:38729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:36.086894 22485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:36.088342 22485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:36.102195 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:36.102378 22501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:36.102313 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:36.103168 22485 server_base.cc:1034] running on GCE node
I20260809 06:50:36.103880 22485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:36.104872 22485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:36.106083 22485 hybrid_clock.cc:648] HybridClock initialized: now 1786258236106005 us; error 69 us; skew 500 ppm
I20260809 06:50:36.106410 22485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:36 dist-test-slave-jpj1 krb5kdc[21929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:36.109604 22485 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:50:36.111171 22485 webserver.cc:460] Webserver started at http://127.27.209.195:35453/ using document root <none> and password file <none>
I20260809 06:50:36.111940 22485 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:36.112071 22485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:36.112401 22485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:36.115497 22485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "ce235b8ad2bf410e9da111427e886c3a"
format_stamp: "Formatted at 2026-08-09 06:50:36 on dist-test-slave-jpj1"
server_key: "c7bbee34ebce9f7781c0dc6482cca7a1"
server_key_iv: "694f2b9779069a7613e8e3b0d934bd19"
server_key_version: "encryptionkey@0"
I20260809 06:50:36.116336 22485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "ce235b8ad2bf410e9da111427e886c3a"
format_stamp: "Formatted at 2026-08-09 06:50:36 on dist-test-slave-jpj1"
server_key: "c7bbee34ebce9f7781c0dc6482cca7a1"
server_key_iv: "694f2b9779069a7613e8e3b0d934bd19"
server_key_version: "encryptionkey@0"
I20260809 06:50:36.121793 22485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 06:50:36.125432 22513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:36.126852 22485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:50:36.127014 22485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "ce235b8ad2bf410e9da111427e886c3a"
format_stamp: "Formatted at 2026-08-09 06:50:36 on dist-test-slave-jpj1"
server_key: "c7bbee34ebce9f7781c0dc6482cca7a1"
server_key_iv: "694f2b9779069a7613e8e3b0d934bd19"
server_key_version: "encryptionkey@0"
I20260809 06:50:36.127219 22485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:36.144956 22485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:36.149092 22485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:36.149401 22485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:36.150476 22485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:36.151924 22485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:36.152004 22485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:36.152128 22485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:36.152184 22485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:36.173961 22292 heartbeater.cc:503] Master 127.27.209.254:38729 was elected leader, sending a full tablet report...
I20260809 06:50:36.184298 22485 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:34419
I20260809 06:50:36.184342 22698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:34419 every 8 connection(s)
I20260809 06:50:36.185593 22485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:36.191998 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 22485
I20260809 06:50:36.192198 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:50:36.192773 28487 external_mini_cluster.cc:1428] Setting key ed91c41ec1e4b55dabeaf64ea8e68d8b
Aug 09 06:50:36 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:36.206446 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.188424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:53089 (local address 127.27.209.254:38729)
0809 06:50:36.188942 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.188948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.189841 (+ 893us) server_negotiation.cc:407] Connection header received
0809 06:50:36.191739 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.191743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.191813 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.191910 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:36.194024 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.194953 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.196800 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.197231 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.197332 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.200749 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:36.200754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:36.200758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:36.200791 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:36.203613 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.204298 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.204303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.204305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.204427 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.204838 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.204842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.204844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.205122 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:36.205248 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.205976 (+ 728us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.206166 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
I20260809 06:50:36.207505 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.188899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:38729 (local address 127.27.209.195:53089)
0809 06:50:36.189678 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:36.189724 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:36.191408 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:36.192194 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:36.192207 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:36.192958 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:36.193832 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.193862 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.195111 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.195115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.196653 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.196666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.197430 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.197435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.197667 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.198309 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:36.198339 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:36.200529 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:36.203737 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:36.203743 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:36.203756 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:36.204189 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:36.204509 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:36.204512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:36.204515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:36.204728 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:36.205263 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:36.205269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:36.205631 (+ 362us) client_negotiation.cc:769] Sending connection context
0809 06:50:36.205865 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 06:50:36.206278 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":271,"threads_started":1}
I20260809 06:50:36.208997 22699 heartbeater.cc:347] Connected to a master server at 127.27.209.254:38729
I20260809 06:50:36.209309 22699 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:36.210057 22699 heartbeater.cc:511] Master 127.27.209.254:38729 requested a full tablet report, sending...
I20260809 06:50:36.211469 22003 ts_manager.cc:194] Registered new tserver with Master: ce235b8ad2bf410e9da111427e886c3a (127.27.209.195:34419)
I20260809 06:50:36.212342 22003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:53089
I20260809 06:50:36.221112 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:50:36.240361 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.227802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.27.209.254:38729)
0809 06:50:36.228021 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.228028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.228354 (+ 326us) server_negotiation.cc:407] Connection header received
0809 06:50:36.228527 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.228531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.228613 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.228701 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:36.230175 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.230967 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.232491 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.232980 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.233085 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.235399 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:36.235403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:36.235406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:36.235440 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:36.237869 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.238553 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.238557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.238559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.238699 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.239008 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.239011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.239013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.239356 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:36.239590 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.239880 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.240065 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:50:36.244473 22003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:50:36.251485 22003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:50:36.285780 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.264202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.27.209.193:35641)
0809 06:50:36.276281 (+ 12079us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.276291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.276325 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:50:36.276433 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.276441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.276741 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.277727 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:36.277975 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.279664 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.282378 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.284389 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.285019 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.285191 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.285266 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.285433 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11873,"thread_start_us":11692,"threads_started":1}
I20260809 06:50:36.285840 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.264456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.27.209.194:38979)
0809 06:50:36.274553 (+ 10097us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.274563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.276908 (+ 2345us) server_negotiation.cc:407] Connection header received
0809 06:50:36.277028 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.277036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.277419 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.277618 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:36.278592 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.280199 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.283111 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.284285 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.284439 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.284622 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.285353 (+ 731us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.285519 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9925,"thread_start_us":9794,"threads_started":1}
I20260809 06:50:36.285872 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.264178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38979 (local address 127.0.0.1:48168)
0809 06:50:36.276582 (+ 12404us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:36.276604 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:36.276754 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:36.277888 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:36.277894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:36.277921 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:36.278396 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.278408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.280354 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.280359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.282956 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.282969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.284773 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.284780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.284931 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.285051 (+ 120us) client_negotiation.cc:769] Sending connection context
0809 06:50:36.285179 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 06:50:36.285405 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12256,"mutex_wait_us":70,"thread_start_us":10542,"threads_started":1}
I20260809 06:50:36.286499 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.264108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:35641 (local address 127.0.0.1:39646)
0809 06:50:36.275184 (+ 11076us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:36.275235 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:36.275465 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:36.277013 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:36.277023 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:36.277059 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:36.277501 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.277515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.279829 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.279834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.282210 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.282226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.284623 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.284630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.284883 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.285051 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 06:50:36.285179 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 06:50:36.286140 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10755,"mutex_wait_us":803,"thread_start_us":12148,"threads_started":1}
I20260809 06:50:36.288395 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.276619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:34419 (local address 127.0.0.1:55570)
0809 06:50:36.280388 (+ 3769us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:36.280407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:36.280538 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:36.281320 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:36.281325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:36.281352 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:36.281771 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.281782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.283723 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.283729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.285767 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.285784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.287649 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.287656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.287831 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.287864 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:50:36.287955 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 06:50:36.288072 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3637,"thread_start_us":3437,"threads_started":1}
I20260809 06:50:36.289214 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.276207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.27.209.195:34419)
0809 06:50:36.280017 (+ 3810us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.280025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.280665 (+ 640us) server_negotiation.cc:407] Connection header received
0809 06:50:36.280791 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.280799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.281019 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.281211 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:36.281960 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.283471 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.285976 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.287130 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.287321 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.287478 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.288756 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.288916 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3605,"thread_start_us":3457,"threads_started":1}
I20260809 06:50:36.290346 22395 tablet_service.cc:1467] Processing CreateTablet for tablet 1798740c75e64138acce8476cf0d474f (DEFAULT_TABLE table=test-table [id=5dd7add58d8845ad9d32ecadf81b95fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:36.290320 22203 tablet_service.cc:1467] Processing CreateTablet for tablet 1798740c75e64138acce8476cf0d474f (DEFAULT_TABLE table=test-table [id=5dd7add58d8845ad9d32ecadf81b95fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:36.291651 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1798740c75e64138acce8476cf0d474f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:36.291818 22203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1798740c75e64138acce8476cf0d474f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:36.292862 22594 tablet_service.cc:1467] Processing CreateTablet for tablet 1798740c75e64138acce8476cf0d474f (DEFAULT_TABLE table=test-table [id=5dd7add58d8845ad9d32ecadf81b95fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:36.294278 22594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1798740c75e64138acce8476cf0d474f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:36.303309 22725 tablet_bootstrap.cc:492] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Bootstrap starting.
I20260809 06:50:36.306533 22727 tablet_bootstrap.cc:492] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Bootstrap starting.
I20260809 06:50:36.307605 22725 tablet_bootstrap.cc:654] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:36.308132 22729 tablet_bootstrap.cc:492] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Bootstrap starting.
I20260809 06:50:36.309373 22725 log.cc:824] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:36.310175 22727 tablet_bootstrap.cc:654] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:36.311060 22729 tablet_bootstrap.cc:654] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:36.311295 22727 log.cc:824] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:36.312212 22729 log.cc:824] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:36.312713 22725 tablet_bootstrap.cc:492] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: No bootstrap required, opened a new log
I20260809 06:50:36.313072 22725 ts_tablet_manager.cc:1397] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 06:50:36.315523 22727 tablet_bootstrap.cc:492] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: No bootstrap required, opened a new log
I20260809 06:50:36.315882 22727 ts_tablet_manager.cc:1397] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260809 06:50:36.317490 22729 tablet_bootstrap.cc:492] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: No bootstrap required, opened a new log
I20260809 06:50:36.317723 22729 ts_tablet_manager.cc:1397] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260809 06:50:36.323212 22725 raft_consensus.cc:348] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.323693 22725 raft_consensus.cc:374] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:36.323837 22725 raft_consensus.cc:729] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3d6122a5994af69860e2ea1a1432e5, State: Initialized, Role: FOLLOWER
I20260809 06:50:36.324470 22725 consensus_queue.cc:260] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.325635 22480 heartbeater.cc:503] Master 127.27.209.254:38729 was elected leader, sending a full tablet report...
I20260809 06:50:36.325579 22729 raft_consensus.cc:348] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.326136 22729 raft_consensus.cc:374] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:36.326004 22727 raft_consensus.cc:348] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.326300 22725 ts_tablet_manager.cc:1428] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260809 06:50:36.326293 22729 raft_consensus.cc:729] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce235b8ad2bf410e9da111427e886c3a, State: Initialized, Role: FOLLOWER
I20260809 06:50:36.326417 22727 raft_consensus.cc:374] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:36.326498 22727 raft_consensus.cc:729] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e1e28bce6443f588dc35a96297d3b4, State: Initialized, Role: FOLLOWER
I20260809 06:50:36.326990 22729 consensus_queue.cc:260] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.327106 22727 consensus_queue.cc:260] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.328120 22699 heartbeater.cc:503] Master 127.27.209.254:38729 was elected leader, sending a full tablet report...
I20260809 06:50:36.328670 22729 ts_tablet_manager.cc:1428] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260809 06:50:36.330403 22727 ts_tablet_manager.cc:1428] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
W20260809 06:50:36.394837 22294 tablet.cc:2369] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:50:36.406639 22481 tablet.cc:2369] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:50:36.437337 22700 tablet.cc:2369] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:36.534940 22735 raft_consensus.cc:482] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:36.535284 22735 raft_consensus.cc:504] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.537123 22735 leader_election.cc:288] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e1e28bce6443f588dc35a96297d3b4 (127.27.209.193:35641), 4a3d6122a5994af69860e2ea1a1432e5 (127.27.209.194:38979)
I20260809 06:50:36.542784 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.537597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:39077 (local address 127.27.209.193:35641)
0809 06:50:36.537816 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.537821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.538047 (+ 226us) server_negotiation.cc:407] Connection header received
0809 06:50:36.538235 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.538239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.538301 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.538378 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:36.538805 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.539513 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.541137 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.541925 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.542072 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.542144 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.542395 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.542524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:50:36.542873 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.537572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:35641 (local address 127.27.209.195:39077)
0809 06:50:36.537943 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:36.537961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:36.538097 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:36.538414 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:36.538419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:36.538445 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:36.538666 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.538676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.539664 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.539670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.540985 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.540998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.542070 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.542076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.542249 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.542273 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:50:36.542352 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 06:50:36.542492 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":4,"spinlock_wait_cycles":24320}
I20260809 06:50:36.543480 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.537632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:38979 (local address 127.27.209.195:42553)
0809 06:50:36.538386 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:36.538400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:36.538544 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:36.538849 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:36.538853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:36.538876 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:36.539187 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.539205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.540266 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.540270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.541629 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:36.541641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.542748 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.542754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:36.542919 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.542956 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 06:50:36.543082 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 06:50:36.543206 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":25,"thread_start_us":454,"threads_started":1}
I20260809 06:50:36.543517 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.537704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:42553 (local address 127.27.209.194:38979)
0809 06:50:36.537915 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.537920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.538495 (+ 575us) server_negotiation.cc:407] Connection header received
0809 06:50:36.538676 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.538680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.538749 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.538821 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:36.539356 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.540106 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.541780 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.542534 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.542682 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.542763 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.543153 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.543286 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:50:36.544096 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1798740c75e64138acce8476cf0d474f" candidate_uuid: "ce235b8ad2bf410e9da111427e886c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e1e28bce6443f588dc35a96297d3b4" is_pre_election: true
I20260809 06:50:36.544412 22421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1798740c75e64138acce8476cf0d474f" candidate_uuid: "ce235b8ad2bf410e9da111427e886c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3d6122a5994af69860e2ea1a1432e5" is_pre_election: true
I20260809 06:50:36.544638 22232 raft_consensus.cc:2454] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce235b8ad2bf410e9da111427e886c3a in term 0.
I20260809 06:50:36.544936 22421 raft_consensus.cc:2454] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce235b8ad2bf410e9da111427e886c3a in term 0.
I20260809 06:50:36.545410 22516 leader_election.cc:302] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e1e28bce6443f588dc35a96297d3b4, ce235b8ad2bf410e9da111427e886c3a; no voters:
I20260809 06:50:36.545910 22735 raft_consensus.cc:2781] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:50:36.546038 22735 raft_consensus.cc:482] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:50:36.546135 22735 raft_consensus.cc:3037] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:36.548401 22735 raft_consensus.cc:504] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.549033 22735 leader_election.cc:288] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [CANDIDATE]: Term 1 election: Requested vote from peers 14e1e28bce6443f588dc35a96297d3b4 (127.27.209.193:35641), 4a3d6122a5994af69860e2ea1a1432e5 (127.27.209.194:38979)
I20260809 06:50:36.549494 22232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1798740c75e64138acce8476cf0d474f" candidate_uuid: "ce235b8ad2bf410e9da111427e886c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e1e28bce6443f588dc35a96297d3b4"
I20260809 06:50:36.549502 22421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1798740c75e64138acce8476cf0d474f" candidate_uuid: "ce235b8ad2bf410e9da111427e886c3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3d6122a5994af69860e2ea1a1432e5"
I20260809 06:50:36.549708 22421 raft_consensus.cc:3037] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:36.549739 22232 raft_consensus.cc:3037] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:36.552073 22232 raft_consensus.cc:2454] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce235b8ad2bf410e9da111427e886c3a in term 1.
I20260809 06:50:36.552431 22516 leader_election.cc:302] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e1e28bce6443f588dc35a96297d3b4, ce235b8ad2bf410e9da111427e886c3a; no voters:
I20260809 06:50:36.552439 22421 raft_consensus.cc:2454] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce235b8ad2bf410e9da111427e886c3a in term 1.
I20260809 06:50:36.552685 22735 raft_consensus.cc:2781] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:36.553180 22735 raft_consensus.cc:686] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 1 LEADER]: Becoming Leader. State: Replica: ce235b8ad2bf410e9da111427e886c3a, State: Running, Role: LEADER
I20260809 06:50:36.553718 22735 consensus_queue.cc:237] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } }
I20260809 06:50:36.558373 22003 catalog_manager.cc:5495] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a reported cstate change: term changed from 0 to 1, leader changed from <none> to ce235b8ad2bf410e9da111427e886c3a (127.27.209.195). New cstate: current_term: 1 leader_uuid: "ce235b8ad2bf410e9da111427e886c3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce235b8ad2bf410e9da111427e886c3a" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 34419 } health_report { overall_health: HEALTHY } } }
I20260809 06:50:36.588222 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.583139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.27.209.195:34419)
0809 06:50:36.583391 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.583396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.583417 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:50:36.583623 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.583627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.583725 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.583857 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:50:36.584186 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.584894 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.586230 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.586600 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.586692 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.587116 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:50:36.587700 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:50:36.587813 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.587888 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.587951 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260809 06:50:36.596354 22232 raft_consensus.cc:1261] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 1 FOLLOWER]: Refusing update from remote peer ce235b8ad2bf410e9da111427e886c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:36.596354 22421 raft_consensus.cc:1261] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 1 FOLLOWER]: Refusing update from remote peer ce235b8ad2bf410e9da111427e886c3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:36.597285 22738 consensus_queue.cc:1035] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3d6122a5994af69860e2ea1a1432e5" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 38979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:36.597622 22735 consensus_queue.cc:1035] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e1e28bce6443f588dc35a96297d3b4" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 35641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:36.624572 22002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:36.624894 22002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:36.628906 22002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 921fe938412343f69a7851c0e2540661: Sending DeleteTablet for 3 replicas of tablet 1798740c75e64138acce8476cf0d474f
I20260809 06:50:36.630179 22203 tablet_service.cc:1514] Processing DeleteTablet for tablet 1798740c75e64138acce8476cf0d474f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:36 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:39646
I20260809 06:50:36.630230 22395 tablet_service.cc:1514] Processing DeleteTablet for tablet 1798740c75e64138acce8476cf0d474f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:36 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:48168
I20260809 06:50:36.631223 22594 tablet_service.cc:1514] Processing DeleteTablet for tablet 1798740c75e64138acce8476cf0d474f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:36 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:55570
I20260809 06:50:36.631678 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 22091
I20260809 06:50:36.641186 22765 tablet_replica.cc:330] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: stopping tablet replica
I20260809 06:50:36.642525 22765 raft_consensus.cc:2229] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:36.644117 22765 raft_consensus.cc:2258] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:36.644785 22766 tablet_replica.cc:330] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: stopping tablet replica
I20260809 06:50:36.645241 22766 raft_consensus.cc:2229] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:36.645857 22766 raft_consensus.cc:2258] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:36.647071 22765 ts_tablet_manager.cc:1905] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:36.647149 22767 tablet_replica.cc:330] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: stopping tablet replica
I20260809 06:50:36.647581 22767 raft_consensus.cc:2229] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:50:36.648371 22767 raft_consensus.cc:2258] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:36.650014 22766 ts_tablet_manager.cc:1905] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:50:36 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:50:36.656685 22767 ts_tablet_manager.cc:1905] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:36.657627 22766 ts_tablet_manager.cc:1918] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:36.657831 22766 log.cc:1196] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/1798740c75e64138acce8476cf0d474f
I20260809 06:50:36.658422 22766 ts_tablet_manager.cc:1939] T 1798740c75e64138acce8476cf0d474f P 14e1e28bce6443f588dc35a96297d3b4: Deleting consensus metadata
I20260809 06:50:36.660624 21986 catalog_manager.cc:4841] TS 14e1e28bce6443f588dc35a96297d3b4 (127.27.209.193:35641): tablet 1798740c75e64138acce8476cf0d474f (table test-table [id=5dd7add58d8845ad9d32ecadf81b95fd]) successfully deleted
I20260809 06:50:36.662961 22765 ts_tablet_manager.cc:1918] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:36.663146 22765 log.cc:1196] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/1798740c75e64138acce8476cf0d474f
I20260809 06:50:36.663738 22765 ts_tablet_manager.cc:1939] T 1798740c75e64138acce8476cf0d474f P 4a3d6122a5994af69860e2ea1a1432e5: Deleting consensus metadata
I20260809 06:50:36.665443 21987 catalog_manager.cc:4841] TS 4a3d6122a5994af69860e2ea1a1432e5 (127.27.209.194:38979): tablet 1798740c75e64138acce8476cf0d474f (table test-table [id=5dd7add58d8845ad9d32ecadf81b95fd]) successfully deleted
I20260809 06:50:36.669021 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.632919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.27.209.193:35641)
0809 06:50:36.633121 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.633127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.644071 (+ 10944us) server_negotiation.cc:407] Connection header received
0809 06:50:36.644303 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.644308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.644402 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.644522 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:36.646389 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.647570 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.649597 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.650244 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.650349 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.661335 (+ 10986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:36.661350 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:36.661368 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:36.661421 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:36.665298 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.666264 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.666274 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.666278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.666449 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.666936 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.666941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.666945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.667440 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:36.667770 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.668389 (+ 619us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.668630 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:50:36.670701 22287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:36.671233 22767 ts_tablet_manager.cc:1918] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:36.671471 22767 log.cc:1196] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/1798740c75e64138acce8476cf0d474f
I20260809 06:50:36.672467 22767 ts_tablet_manager.cc:1939] T 1798740c75e64138acce8476cf0d474f P ce235b8ad2bf410e9da111427e886c3a: Deleting consensus metadata
I20260809 06:50:36.674913 21987 catalog_manager.cc:4841] TS ce235b8ad2bf410e9da111427e886c3a (127.27.209.195:34419): tablet 1798740c75e64138acce8476cf0d474f (table test-table [id=5dd7add58d8845ad9d32ecadf81b95fd]) successfully deleted
I20260809 06:50:36.763377 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 22091
I20260809 06:50:36.793320 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 22300
Aug 09 06:50:36 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:50:36.809708 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.794124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.27.209.194:38979)
0809 06:50:36.794310 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.794316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.794394 (+ 78us) server_negotiation.cc:407] Connection header received
0809 06:50:36.794593 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.794597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.794679 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.794770 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:36.796099 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.797023 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.798315 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.798708 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.798817 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.803444 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:36.803464 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:36.803486 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:36.803554 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:36.806906 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.807627 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.807635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.807637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.807776 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.808130 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.808134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.808136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.808531 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:36.808818 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.809146 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.809381 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 06:50:36.810452 22471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:36.898722 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 22300
I20260809 06:50:36.928706 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 22485
Aug 09 06:50:36 dist-test-slave-jpj1 krb5kdc[21929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:50:36.947463 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:36.929585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.27.209.195:34419)
0809 06:50:36.929783 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:36.929788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:36.929809 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:50:36.929962 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:36.929968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:36.930050 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:36.930151 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:36.931514 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.932417 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.934454 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:36.934925 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:36.935029 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:36.940440 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:36.940466 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:36.940489 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:36.940544 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:36.944524 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.945235 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.945245 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.945248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.945459 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:36.945825 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:36.945834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:36.945838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:36.946297 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:36.946658 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:36.946825 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:50:36.947068 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 06:50:36.948269 22694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:37.070376 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 22485
I20260809 06:50:37.096524 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 21951
I20260809 06:50:37.097711 22044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:37.207085 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 21951
2026-08-09T06:50:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4776 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T06:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:37Z Disabled control of system clock
I20260809 06:50:37.300560 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:43109
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:43109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:50:37.642563 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:37.642858 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:37.642939 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:37.650064 22784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:37.650171 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:37.650224 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:37.650288 22784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:37.650332 22784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:37.659276 22784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:43109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:43109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:37.660548 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:37.662307 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:37.674314 22794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:37.674342 22797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:37.674407 22793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:37.674968 22784 server_base.cc:1034] running on GCE node
I20260809 06:50:37.675866 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:37.677294 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:37.678493 22784 hybrid_clock.cc:648] HybridClock initialized: now 1786258237678440 us; error 46 us; skew 500 ppm
I20260809 06:50:37.678844 22784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:37.681033 22784 webserver.cc:460] Webserver started at http://127.27.209.254:40933/ using document root <none> and password file <none>
I20260809 06:50:37.681715 22784 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:37.681818 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:37.682125 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:37.685191 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "fc964a7a9f12413580f26e44780b5eb4"
format_stamp: "Formatted at 2026-08-09 06:50:37 on dist-test-slave-jpj1"
server_key: "dbdd653dc006873eb7da2372f252fcee"
server_key_iv: "1be64e6b7cb71ec3b11a64b611af30bb"
server_key_version: "encryptionkey@0"
I20260809 06:50:37.686007 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "fc964a7a9f12413580f26e44780b5eb4"
format_stamp: "Formatted at 2026-08-09 06:50:37 on dist-test-slave-jpj1"
server_key: "dbdd653dc006873eb7da2372f252fcee"
server_key_iv: "1be64e6b7cb71ec3b11a64b611af30bb"
server_key_version: "encryptionkey@0"
I20260809 06:50:37.691303 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:50:37.694741 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:37.696120 22784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:37.696288 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "fc964a7a9f12413580f26e44780b5eb4"
format_stamp: "Formatted at 2026-08-09 06:50:37 on dist-test-slave-jpj1"
server_key: "dbdd653dc006873eb7da2372f252fcee"
server_key_iv: "1be64e6b7cb71ec3b11a64b611af30bb"
server_key_version: "encryptionkey@0"
I20260809 06:50:37.696484 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:37.718079 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:37.718899 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:37.719179 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:37.736971 22784 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:43109
I20260809 06:50:37.736989 22896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:43109 every 8 connection(s)
I20260809 06:50:37.738301 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:37.742179 22900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:37.747061 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 22784
I20260809 06:50:37.747248 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:37.747743 28487 external_mini_cluster.cc:1428] Setting key f1f74f17ea2cad149df00958d878d6c4
I20260809 06:50:37.753165 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4: Bootstrap starting.
I20260809 06:50:37.756770 22900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:37.757985 22900 log.cc:824] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:37.761153 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4: No bootstrap required, opened a new log
I20260809 06:50:37.761655 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:37.749559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.27.209.254:43109)
0809 06:50:37.750595 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:37.750625 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:37.750671 (+ 46us) server_negotiation.cc:407] Connection header received
0809 06:50:37.752394 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:37.752419 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:37.752863 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:37.753170 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:37.753784 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:37.755099 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:37.757138 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:37.757860 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:37.758006 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:37.759316 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:37.759327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:37.759346 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:37.759928 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:37.760001 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:37.760022 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:37.760108 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:37.760365 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 06:50:37.760693 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":374,"threads_started":1}
I20260809 06:50:37.768965 22900 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc964a7a9f12413580f26e44780b5eb4" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43109 } }
I20260809 06:50:37.769322 22900 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:37.769423 22900 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc964a7a9f12413580f26e44780b5eb4, State: Initialized, Role: FOLLOWER
I20260809 06:50:37.769999 22900 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc964a7a9f12413580f26e44780b5eb4" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43109 } }
I20260809 06:50:37.770221 22900 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:37.770350 22900 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:37.770515 22900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:37.772964 22900 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc964a7a9f12413580f26e44780b5eb4" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43109 } }
I20260809 06:50:37.773450 22900 leader_election.cc:302] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc964a7a9f12413580f26e44780b5eb4; no voters:
I20260809 06:50:37.773998 22900 leader_election.cc:288] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:37.774128 22905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:37.774766 22905 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [term 1 LEADER]: Becoming Leader. State: Replica: fc964a7a9f12413580f26e44780b5eb4, State: Running, Role: LEADER
I20260809 06:50:37.775290 22905 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc964a7a9f12413580f26e44780b5eb4" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43109 } }
I20260809 06:50:37.776280 22900 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:37.777974 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc964a7a9f12413580f26e44780b5eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc964a7a9f12413580f26e44780b5eb4" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43109 } } }
I20260809 06:50:37.778288 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc964a7a9f12413580f26e44780b5eb4. Latest consensus state: current_term: 1 leader_uuid: "fc964a7a9f12413580f26e44780b5eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc964a7a9f12413580f26e44780b5eb4" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 43109 } } }
I20260809 06:50:37.778528 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:37.778606 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:37.780226 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:37.785308 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:37.793778 22914 catalog_manager.cc:1347] Generated new cluster ID: ff99ecad826f49ebb0f36118d0f77ed0
I20260809 06:50:37.793887 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:37.825292 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:37.827147 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:37.835726 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4: Generated new TSK 0
I20260809 06:50:37.836795 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:50:37.846004 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43109
--builtin_ntp_servers=127.27.209.212:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:50:38.206525 22935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:38.206875 22935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:38.207015 22935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:38.214308 22935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:38.214416 22935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:38.214561 22935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:38.223567 22935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:38.224819 22935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:38.226613 22935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:38.239923 22945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:38.239948 22946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:38.239876 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:38.240690 22935 server_base.cc:1034] running on GCE node
I20260809 06:50:38.241269 22935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:38.242192 22935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:38.243414 22935 hybrid_clock.cc:648] HybridClock initialized: now 1786258238243355 us; error 53 us; skew 500 ppm
I20260809 06:50:38.243755 22935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:38.246014 22935 webserver.cc:460] Webserver started at http://127.27.209.193:35231/ using document root <none> and password file <none>
I20260809 06:50:38.246742 22935 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:38.246860 22935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:38.247169 22935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:38.250211 22935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "d92e6a476e824ed2895ea24047386268"
format_stamp: "Formatted at 2026-08-09 06:50:38 on dist-test-slave-jpj1"
server_key: "8ded6ee1f6d4abd754c528d73db9a0bf"
server_key_iv: "50c0f6a5bc9b5c389d6ae8834fa16617"
server_key_version: "encryptionkey@0"
I20260809 06:50:38.251029 22935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "d92e6a476e824ed2895ea24047386268"
format_stamp: "Formatted at 2026-08-09 06:50:38 on dist-test-slave-jpj1"
server_key: "8ded6ee1f6d4abd754c528d73db9a0bf"
server_key_iv: "50c0f6a5bc9b5c389d6ae8834fa16617"
server_key_version: "encryptionkey@0"
I20260809 06:50:38.256150 22935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 06:50:38.259784 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:38.261107 22935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:50:38.261271 22935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "d92e6a476e824ed2895ea24047386268"
format_stamp: "Formatted at 2026-08-09 06:50:38 on dist-test-slave-jpj1"
server_key: "8ded6ee1f6d4abd754c528d73db9a0bf"
server_key_iv: "50c0f6a5bc9b5c389d6ae8834fa16617"
server_key_version: "encryptionkey@0"
I20260809 06:50:38.261472 22935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:38.310503 22935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:38.311333 22935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:38.311645 22935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:38.312628 22935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:38.314085 22935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:38.314180 22935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:38.314291 22935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:38.314363 22935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:38.344168 22935 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:41405
I20260809 06:50:38.344238 23125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:41405 every 8 connection(s)
I20260809 06:50:38.345455 22935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:38.350317 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 22935
I20260809 06:50:38.350483 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:50:38.350890 28487 external_mini_cluster.cc:1428] Setting key a7c744cbdcfe81fd7eef02fd17938a95
I20260809 06:50:38.353662 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43109
--builtin_ntp_servers=127.27.209.212:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:50:38.357841 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:38.347923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:55343 (local address 127.27.209.254:43109)
0809 06:50:38.348344 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:38.348349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:38.349459 (+ 1110us) server_negotiation.cc:407] Connection header received
0809 06:50:38.351354 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:38.351361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:38.351442 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:38.351563 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:38.352875 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.353555 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.355143 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.355580 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.355698 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:38.357003 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:38.357008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:38.357011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:38.357029 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:38.357068 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:38.357088 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:38.357162 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:38.357377 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 06:50:38.357479 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":217,"threads_started":1}
I20260809 06:50:38.358914 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:38.348422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:43109 (local address 127.27.209.193:55343)
0809 06:50:38.349301 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:38.349344 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:38.351088 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:38.351851 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:38.351863 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:38.351979 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:38.352710 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:38.352731 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.353708 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.353712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:38.354995 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:38.355008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.355798 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.355807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:38.356019 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:38.356679 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:38.356681 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:38.356723 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:38.356805 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:38.357178 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:38.357191 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:38.357197 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 06:50:38.357353 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 06:50:38.357766 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":252,"threads_started":1}
I20260809 06:50:38.360472 23126 heartbeater.cc:347] Connected to a master server at 127.27.209.254:43109
I20260809 06:50:38.360767 23126 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:38.361503 23126 heartbeater.cc:511] Master 127.27.209.254:43109 requested a full tablet report, sending...
I20260809 06:50:38.363736 22831 ts_manager.cc:194] Registered new tserver with Master: d92e6a476e824ed2895ea24047386268 (127.27.209.193:41405)
I20260809 06:50:38.365672 22831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.193:55343
W20260809 06:50:38.683472 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:38.683794 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:38.683889 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:38.691051 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:38.691140 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:38.691242 23135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:38.700102 23135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:38.701407 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:38.703173 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:38.715822 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:38.715857 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:38.715929 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:38.716701 23135 server_base.cc:1034] running on GCE node
I20260809 06:50:38.717267 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:38.718156 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:38.719355 23135 hybrid_clock.cc:648] HybridClock initialized: now 1786258238719304 us; error 46 us; skew 500 ppm
I20260809 06:50:38.719743 23135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:38.721915 23135 webserver.cc:460] Webserver started at http://127.27.209.194:35591/ using document root <none> and password file <none>
I20260809 06:50:38.722535 23135 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:38.722621 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:38.722891 23135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:38.725628 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "b4821327ac114fafbebde48333d8bdda"
format_stamp: "Formatted at 2026-08-09 06:50:38 on dist-test-slave-jpj1"
server_key: "eae35825d608fedeef87d1155eb147e7"
server_key_iv: "10a46fb2ef0eafe7436aaf072b18644f"
server_key_version: "encryptionkey@0"
I20260809 06:50:38.726356 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "b4821327ac114fafbebde48333d8bdda"
format_stamp: "Formatted at 2026-08-09 06:50:38 on dist-test-slave-jpj1"
server_key: "eae35825d608fedeef87d1155eb147e7"
server_key_iv: "10a46fb2ef0eafe7436aaf072b18644f"
server_key_version: "encryptionkey@0"
I20260809 06:50:38.731209 23135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:50:38.734551 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:38.735828 23135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:38.735957 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "b4821327ac114fafbebde48333d8bdda"
format_stamp: "Formatted at 2026-08-09 06:50:38 on dist-test-slave-jpj1"
server_key: "eae35825d608fedeef87d1155eb147e7"
server_key_iv: "10a46fb2ef0eafe7436aaf072b18644f"
server_key_version: "encryptionkey@0"
I20260809 06:50:38.736135 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:38.751389 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:38.752224 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:38.752504 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:38.753422 23135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:38.754810 23135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:38.754909 23135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:38.755018 23135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:38.755087 23135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:38.785715 23135 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:41251
I20260809 06:50:38.785770 23352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:41251 every 8 connection(s)
I20260809 06:50:38.787002 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:38.792683 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 23135
I20260809 06:50:38.792876 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:50:38.793434 28487 external_mini_cluster.cc:1428] Setting key c0c9720ffc22d4f4c5adfb3f749b6dcd
I20260809 06:50:38.796733 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43109
--builtin_ntp_servers=127.27.209.212:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:50:38.800406 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:38.789463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:34303 (local address 127.27.209.254:43109)
0809 06:50:38.789658 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:38.789663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:38.790970 (+ 1307us) server_negotiation.cc:407] Connection header received
0809 06:50:38.792941 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:38.792948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:38.793032 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:38.793125 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:38.794435 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.795475 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.797707 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.798102 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.798209 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:38.799733 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:38.799737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:38.799741 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:38.799759 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:38.799791 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:38.799804 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:38.799872 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:38.800084 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 06:50:38.800173 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:50:38.801659 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:38.789926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:43109 (local address 127.27.209.194:34303)
0809 06:50:38.790821 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:38.790865 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:38.792566 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:38.793385 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:38.793396 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:38.793510 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:38.794260 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:38.794287 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.795670 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.795675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:38.797556 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:38.797573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:38.798325 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:38.798332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:38.798597 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:38.799329 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:38.799333 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:38.799384 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:38.799472 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:38.799890 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:38.799901 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:38.799909 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:50:38.800081 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 06:50:38.800446 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260809 06:50:38.802989 23353 heartbeater.cc:347] Connected to a master server at 127.27.209.254:43109
I20260809 06:50:38.803287 23353 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:38.804191 23353 heartbeater.cc:511] Master 127.27.209.254:43109 requested a full tablet report, sending...
I20260809 06:50:38.805745 22831 ts_manager.cc:194] Registered new tserver with Master: b4821327ac114fafbebde48333d8bdda (127.27.209.194:41251)
I20260809 06:50:38.806536 22831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.194:34303
W20260809 06:50:39.128551 23359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:39.128820 23359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:39.128911 23359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:39.135886 23359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:39.135980 23359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:39.136083 23359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:39.144878 23359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:43109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:39.146057 23359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:39.147806 23359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:39.160765 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:39.160787 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:39.160744 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:39.161545 23359 server_base.cc:1034] running on GCE node
I20260809 06:50:39.162207 23359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:39.163173 23359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:39.164382 23359 hybrid_clock.cc:648] HybridClock initialized: now 1786258239164327 us; error 37 us; skew 500 ppm
I20260809 06:50:39.164793 23359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:39.167088 23359 webserver.cc:460] Webserver started at http://127.27.209.195:45937/ using document root <none> and password file <none>
I20260809 06:50:39.167853 23359 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:39.167950 23359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:39.168239 23359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:39.171111 23359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "435d70e8674c48ccbee7135f1349452c"
format_stamp: "Formatted at 2026-08-09 06:50:39 on dist-test-slave-jpj1"
server_key: "ee15a79fca09f72a05a1eb6360116830"
server_key_iv: "34f487efe8244bf0f40b9deaa0d81a3b"
server_key_version: "encryptionkey@0"
I20260809 06:50:39.171926 23359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "435d70e8674c48ccbee7135f1349452c"
format_stamp: "Formatted at 2026-08-09 06:50:39 on dist-test-slave-jpj1"
server_key: "ee15a79fca09f72a05a1eb6360116830"
server_key_iv: "34f487efe8244bf0f40b9deaa0d81a3b"
server_key_version: "encryptionkey@0"
I20260809 06:50:39.177217 23359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:50:39.180619 23378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:39.182056 23359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:39.182206 23359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "435d70e8674c48ccbee7135f1349452c"
format_stamp: "Formatted at 2026-08-09 06:50:39 on dist-test-slave-jpj1"
server_key: "ee15a79fca09f72a05a1eb6360116830"
server_key_iv: "34f487efe8244bf0f40b9deaa0d81a3b"
server_key_version: "encryptionkey@0"
I20260809 06:50:39.182407 23359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:39.231833 23359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:39.232740 23359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:39.233045 23359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:39.234133 23359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:39.235674 23359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:39.235805 23359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:39.235918 23359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:39.235991 23359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:39.265739 23359 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:46329
I20260809 06:50:39.265789 23548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:46329 every 8 connection(s)
I20260809 06:50:39.267076 23359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:39.276495 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 23359
I20260809 06:50:39.276684 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:50:39.277102 28487 external_mini_cluster.cc:1428] Setting key c43f8db5e023dd002f8bc1494a3b421a
I20260809 06:50:39.280011 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.269595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:41031 (local address 127.27.209.254:43109)
0809 06:50:39.269793 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.269798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.271160 (+ 1362us) server_negotiation.cc:407] Connection header received
0809 06:50:39.273056 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.273061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.273131 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.273217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:39.274191 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.274937 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.276545 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.277022 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.277139 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.278636 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:39.278642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:39.278647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:39.278670 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:39.278707 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:39.278720 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:39.278789 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.279614 (+ 825us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.279704 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:50:39.280864 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.270144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:43109 (local address 127.27.209.195:41031)
0809 06:50:39.271028 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:39.271064 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:39.272822 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:39.273413 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:39.273421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:39.273505 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:39.274065 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.274082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.275053 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.275057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.276400 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.276413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.277262 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.277270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.277493 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.278281 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:39.278284 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:39.278339 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:39.278450 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:39.278814 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:39.278826 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:39.278834 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:50:39.279016 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 06:50:39.279464 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":249,"threads_started":1}
I20260809 06:50:39.282297 23551 heartbeater.cc:347] Connected to a master server at 127.27.209.254:43109
I20260809 06:50:39.282583 23551 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:39.283337 23551 heartbeater.cc:511] Master 127.27.209.254:43109 requested a full tablet report, sending...
I20260809 06:50:39.284698 22831 ts_manager.cc:194] Registered new tserver with Master: 435d70e8674c48ccbee7135f1349452c (127.27.209.195:46329)
I20260809 06:50:39.285473 22831 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.195:41031
I20260809 06:50:39.292016 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:50:39.301138 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.295848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.27.209.254:43109)
0809 06:50:39.296032 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.296036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.296436 (+ 400us) server_negotiation.cc:407] Connection header received
0809 06:50:39.296572 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.296576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.296644 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.296723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:39.297231 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.297882 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.299152 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.299608 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.299719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.300561 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:39.300566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:39.300569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:39.300586 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:39.300615 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:39.300627 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:39.300707 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.300789 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.300861 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:50:39.304477 22831 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:50:39.311131 22831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:50:39.345431 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.322904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:46329 (local address 127.0.0.1:50012)
0809 06:50:39.336064 (+ 13160us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:39.336081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:39.336185 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:39.337744 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:39.337753 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:39.337785 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:39.338221 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.338246 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.340193 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.340198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.342480 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.342495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.344557 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.344563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.344757 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.344814 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 06:50:39.344939 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 06:50:39.345127 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13049,"thread_start_us":12938,"threads_started":1}
I20260809 06:50:39.346351 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.323369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.27.209.195:46329)
0809 06:50:39.337005 (+ 13636us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.337023 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.337057 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:50:39.337159 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.337166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.337476 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.337690 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:39.338465 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.340032 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.342730 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.344024 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.344203 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.344433 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.345859 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.346031 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13461,"thread_start_us":13311,"threads_started":1}
I20260809 06:50:39.349720 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.338099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:41251 (local address 127.0.0.1:38680)
0809 06:50:39.341741 (+ 3642us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:39.341758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:39.341886 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:39.342547 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:39.342552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:39.342578 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:39.342988 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.342999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.345763 (+ 2764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.345768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.347714 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.347728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.348955 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.348961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.349123 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.349155 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:50:39.349242 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 06:50:39.349359 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3507,"mutex_wait_us":25,"thread_start_us":3355,"threads_started":1}
I20260809 06:50:39.349799 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.337002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.27.209.194:41251)
0809 06:50:39.341385 (+ 4383us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.341394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.341871 (+ 477us) server_negotiation.cc:407] Connection header received
0809 06:50:39.342034 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.342041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.342275 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.342482 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:39.343152 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.345599 (+ 2447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.347886 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.348773 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.348917 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.349111 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.349283 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.349436 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4250,"thread_start_us":4131,"threads_started":1}
I20260809 06:50:39.350550 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 0c546334eb38460e8796865beaffda26 (DEFAULT_TABLE table=test-table [id=f8d485596a894c6e9f67d04e9aab5961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:39.347286 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.322936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.27.209.193:41405)
0809 06:50:39.337335 (+ 14399us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.337341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.337367 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:50:39.337444 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.337449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.337617 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.337826 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:39.339948 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.341547 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.343835 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.345291 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.345477 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.345756 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.346738 (+ 982us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.346901 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14242,"thread_start_us":14137,"threads_started":1}
I20260809 06:50:39.351984 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.322876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:41405 (local address 127.0.0.1:38390)
0809 06:50:39.335154 (+ 12278us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:39.335210 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:39.335450 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:39.338087 (+ 2637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:39.338091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:39.338114 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:39.339738 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.339757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.341710 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.341718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.343507 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.343521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.345947 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.345953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.346156 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.346570 (+ 414us) client_negotiation.cc:769] Sending connection context
0809 06:50:39.351501 (+ 4931us) client_negotiation.cc:240] Negotiation successful
0809 06:50:39.351663 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11924,"mutex_wait_us":209,"thread_start_us":13363,"threads_started":1}
I20260809 06:50:39.352167 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c546334eb38460e8796865beaffda26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:39.352942 23240 tablet_service.cc:1467] Processing CreateTablet for tablet 0c546334eb38460e8796865beaffda26 (DEFAULT_TABLE table=test-table [id=f8d485596a894c6e9f67d04e9aab5961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:39.354353 23240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c546334eb38460e8796865beaffda26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:39.355683 23033 tablet_service.cc:1467] Processing CreateTablet for tablet 0c546334eb38460e8796865beaffda26 (DEFAULT_TABLE table=test-table [id=f8d485596a894c6e9f67d04e9aab5961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:39.357151 23033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c546334eb38460e8796865beaffda26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:39.364210 23580 tablet_bootstrap.cc:492] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Bootstrap starting.
I20260809 06:50:39.367520 23580 tablet_bootstrap.cc:654] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:39.368577 23126 heartbeater.cc:503] Master 127.27.209.254:43109 was elected leader, sending a full tablet report...
I20260809 06:50:39.368686 23580 log.cc:824] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:39.369011 23581 tablet_bootstrap.cc:492] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Bootstrap starting.
I20260809 06:50:39.371698 23584 tablet_bootstrap.cc:492] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Bootstrap starting.
I20260809 06:50:39.373514 23581 tablet_bootstrap.cc:654] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:39.374782 23581 log.cc:824] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:39.374902 23584 tablet_bootstrap.cc:654] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:39.376022 23584 log.cc:824] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:39.377836 23581 tablet_bootstrap.cc:492] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: No bootstrap required, opened a new log
I20260809 06:50:39.377887 23580 tablet_bootstrap.cc:492] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: No bootstrap required, opened a new log
I20260809 06:50:39.378080 23581 ts_tablet_manager.cc:1397] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:50:39.378156 23580 ts_tablet_manager.cc:1397] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260809 06:50:39.378963 23584 tablet_bootstrap.cc:492] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: No bootstrap required, opened a new log
I20260809 06:50:39.379199 23584 ts_tablet_manager.cc:1397] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 06:50:39.384984 23581 raft_consensus.cc:348] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.385318 23581 raft_consensus.cc:374] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:39.385406 23581 raft_consensus.cc:729] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4821327ac114fafbebde48333d8bdda, State: Initialized, Role: FOLLOWER
I20260809 06:50:39.386010 23581 consensus_queue.cc:260] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.387727 23353 heartbeater.cc:503] Master 127.27.209.254:43109 was elected leader, sending a full tablet report...
I20260809 06:50:39.388381 23581 ts_tablet_manager.cc:1428] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
I20260809 06:50:39.388401 23580 raft_consensus.cc:348] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.388828 23580 raft_consensus.cc:374] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:39.388939 23580 raft_consensus.cc:729] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 435d70e8674c48ccbee7135f1349452c, State: Initialized, Role: FOLLOWER
I20260809 06:50:39.389532 23580 consensus_queue.cc:260] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.389458 23584 raft_consensus.cc:348] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.389880 23584 raft_consensus.cc:374] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:39.389986 23584 raft_consensus.cc:729] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92e6a476e824ed2895ea24047386268, State: Initialized, Role: FOLLOWER
I20260809 06:50:39.390722 23551 heartbeater.cc:503] Master 127.27.209.254:43109 was elected leader, sending a full tablet report...
I20260809 06:50:39.390674 23584 consensus_queue.cc:260] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.391340 23580 ts_tablet_manager.cc:1428] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260809 06:50:39.394272 23584 ts_tablet_manager.cc:1428] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
W20260809 06:50:39.518937 23552 tablet.cc:2369] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:50:39.539083 23356 tablet.cc:2369] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:39.545218 23588 raft_consensus.cc:482] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:39.545527 23588 raft_consensus.cc:504] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.547355 23588 leader_election.cc:288] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92e6a476e824ed2895ea24047386268 (127.27.209.193:41405), b4821327ac114fafbebde48333d8bdda (127.27.209.194:41251)
I20260809 06:50:39.555733 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.547834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:38355 (local address 127.27.209.193:41405)
0809 06:50:39.548033 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.548040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.548414 (+ 374us) server_negotiation.cc:407] Connection header received
0809 06:50:39.548842 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.548848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.549044 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.549437 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:39.550173 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.551187 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.553581 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.554747 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.554897 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.554969 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.555287 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.555426 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:50:39.556030 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.547753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:41405 (local address 127.27.209.195:38355)
0809 06:50:39.548279 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:39.548306 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:39.548588 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:39.549350 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:39.549357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:39.549389 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:39.549978 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.549992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.551340 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.551344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.553409 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.553424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.554902 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.554908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.555101 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.555134 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:50:39.555231 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 06:50:39.555740 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":26}
I20260809 06:50:39.557242 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.548718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:39985 (local address 127.27.209.194:41251)
0809 06:50:39.548915 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.548920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.550380 (+ 1460us) server_negotiation.cc:407] Connection header received
0809 06:50:39.550646 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.550652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.550741 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.550828 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:39.551444 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.552479 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.555093 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.556224 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.556373 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.556432 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.556848 (+ 416us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.556985 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:50:39.557255 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.548202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:41251 (local address 127.27.209.195:39985)
0809 06:50:39.550276 (+ 2074us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:39.550295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:39.550482 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:39.550866 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:39.550871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:39.550897 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:39.551262 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.551278 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.552647 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.552653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.554833 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:39.554849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.556512 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.556516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:39.556670 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.556701 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:50:39.556800 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 06:50:39.556917 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1914,"mutex_wait_us":42,"thread_start_us":1507,"threads_started":1}
I20260809 06:50:39.557214 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c546334eb38460e8796865beaffda26" candidate_uuid: "435d70e8674c48ccbee7135f1349452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e6a476e824ed2895ea24047386268" is_pre_election: true
I20260809 06:50:39.557875 23063 raft_consensus.cc:2454] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435d70e8674c48ccbee7135f1349452c in term 0.
I20260809 06:50:39.558631 23382 leader_election.cc:302] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435d70e8674c48ccbee7135f1349452c, d92e6a476e824ed2895ea24047386268; no voters:
I20260809 06:50:39.560243 23275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c546334eb38460e8796865beaffda26" candidate_uuid: "435d70e8674c48ccbee7135f1349452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4821327ac114fafbebde48333d8bdda" is_pre_election: true
I20260809 06:50:39.560770 23275 raft_consensus.cc:2454] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 435d70e8674c48ccbee7135f1349452c in term 0.
I20260809 06:50:39.561343 23588 raft_consensus.cc:2781] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:50:39.561483 23588 raft_consensus.cc:482] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:50:39.561570 23588 raft_consensus.cc:3037] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:39.564631 23588 raft_consensus.cc:504] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.565250 23588 leader_election.cc:288] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [CANDIDATE]: Term 1 election: Requested vote from peers d92e6a476e824ed2895ea24047386268 (127.27.209.193:41405), b4821327ac114fafbebde48333d8bdda (127.27.209.194:41251)
I20260809 06:50:39.566052 23063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c546334eb38460e8796865beaffda26" candidate_uuid: "435d70e8674c48ccbee7135f1349452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e6a476e824ed2895ea24047386268"
I20260809 06:50:39.566252 23063 raft_consensus.cc:3037] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:39.566704 23275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c546334eb38460e8796865beaffda26" candidate_uuid: "435d70e8674c48ccbee7135f1349452c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4821327ac114fafbebde48333d8bdda"
I20260809 06:50:39.566905 23275 raft_consensus.cc:3037] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:39.569106 23275 raft_consensus.cc:2454] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435d70e8674c48ccbee7135f1349452c in term 1.
I20260809 06:50:39.569469 23382 leader_election.cc:302] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 435d70e8674c48ccbee7135f1349452c, b4821327ac114fafbebde48333d8bdda; no voters:
I20260809 06:50:39.569720 23063 raft_consensus.cc:2454] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 435d70e8674c48ccbee7135f1349452c in term 1.
I20260809 06:50:39.569762 23588 raft_consensus.cc:2781] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:39.570245 23588 raft_consensus.cc:686] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 1 LEADER]: Becoming Leader. State: Replica: 435d70e8674c48ccbee7135f1349452c, State: Running, Role: LEADER
I20260809 06:50:39.570873 23588 consensus_queue.cc:237] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } }
I20260809 06:50:39.575773 22831 catalog_manager.cc:5495] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c reported cstate change: term changed from 0 to 1, leader changed from <none> to 435d70e8674c48ccbee7135f1349452c (127.27.209.195). New cstate: current_term: 1 leader_uuid: "435d70e8674c48ccbee7135f1349452c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "435d70e8674c48ccbee7135f1349452c" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 46329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 } health_report { overall_health: UNKNOWN } } }
W20260809 06:50:39.598017 23127 tablet.cc:2369] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:39.634927 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.629896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.27.209.195:46329)
0809 06:50:39.630064 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.630069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.630093 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:50:39.630222 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.630226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.630309 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.630436 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:50:39.630770 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.631522 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.632958 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.633368 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.633458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.633971 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:50:39.634479 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:50:39.634624 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.634663 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.634729 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:50:39.643601 23275 raft_consensus.cc:1261] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 1 FOLLOWER]: Refusing update from remote peer 435d70e8674c48ccbee7135f1349452c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:39.643592 23063 raft_consensus.cc:1261] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 1 FOLLOWER]: Refusing update from remote peer 435d70e8674c48ccbee7135f1349452c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:39.644601 23595 consensus_queue.cc:1035] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92e6a476e824ed2895ea24047386268" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:39.645000 23588 consensus_queue.cc:1035] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4821327ac114fafbebde48333d8bdda" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 41251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:39.662894 23600 mvcc.cc:205] Tried to move back new op lower bound from 7316513749568544768 to 7316513749286621184. Current Snapshot: MvccSnapshot[applied={T|T < 7316513749568544768}]
I20260809 06:50:39.664896 23601 mvcc.cc:205] Tried to move back new op lower bound from 7316513749568544768 to 7316513749286621184. Current Snapshot: MvccSnapshot[applied={T|T < 7316513749568544768}]
I20260809 06:50:39.671886 22831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:39.672127 22831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:39.675704 22831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc964a7a9f12413580f26e44780b5eb4: Sending DeleteTablet for 3 replicas of tablet 0c546334eb38460e8796865beaffda26
I20260809 06:50:39.676795 23449 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c546334eb38460e8796865beaffda26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:39 UTC) from {username='slave'} at 127.0.0.1:50012
I20260809 06:50:39.676880 23033 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c546334eb38460e8796865beaffda26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:39 UTC) from {username='slave'} at 127.0.0.1:38390
I20260809 06:50:39.677027 23240 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c546334eb38460e8796865beaffda26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:39 UTC) from {username='slave'} at 127.0.0.1:38680
I20260809 06:50:39.677843 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 22935
I20260809 06:50:39.686899 23613 tablet_replica.cc:330] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: stopping tablet replica
I20260809 06:50:39.687485 23613 raft_consensus.cc:2229] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:39.688202 23613 raft_consensus.cc:2258] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:39.689195 23611 tablet_replica.cc:330] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: stopping tablet replica
I20260809 06:50:39.689939 23614 tablet_replica.cc:330] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: stopping tablet replica
I20260809 06:50:39.690415 23614 raft_consensus.cc:2229] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:39.690929 23614 raft_consensus.cc:2258] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:39.691814 23613 ts_tablet_manager.cc:1905] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:39.692323 23611 raft_consensus.cc:2229] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:50:39.692984 23614 ts_tablet_manager.cc:1905] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:39.693734 23611 raft_consensus.cc:2258] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:39.696864 23611 ts_tablet_manager.cc:1905] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:39.699882 23614 ts_tablet_manager.cc:1918] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:39.700048 23614 log.cc:1196] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/0c546334eb38460e8796865beaffda26
I20260809 06:50:39.700567 23613 ts_tablet_manager.cc:1918] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:39.700678 23614 ts_tablet_manager.cc:1939] T 0c546334eb38460e8796865beaffda26 P b4821327ac114fafbebde48333d8bdda: Deleting consensus metadata
I20260809 06:50:39.700863 23613 log.cc:1196] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/0c546334eb38460e8796865beaffda26
I20260809 06:50:39.701449 23613 ts_tablet_manager.cc:1939] T 0c546334eb38460e8796865beaffda26 P d92e6a476e824ed2895ea24047386268: Deleting consensus metadata
I20260809 06:50:39.702682 22810 catalog_manager.cc:4841] TS b4821327ac114fafbebde48333d8bdda (127.27.209.194:41251): tablet 0c546334eb38460e8796865beaffda26 (table test-table [id=f8d485596a894c6e9f67d04e9aab5961]) successfully deleted
I20260809 06:50:39.703451 22810 catalog_manager.cc:4841] TS d92e6a476e824ed2895ea24047386268 (127.27.209.193:41405): tablet 0c546334eb38460e8796865beaffda26 (table test-table [id=f8d485596a894c6e9f67d04e9aab5961]) successfully deleted
I20260809 06:50:39.703809 23611 ts_tablet_manager.cc:1918] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:39.703955 23611 log.cc:1196] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/0c546334eb38460e8796865beaffda26
I20260809 06:50:39.704403 23611 ts_tablet_manager.cc:1939] T 0c546334eb38460e8796865beaffda26 P 435d70e8674c48ccbee7135f1349452c: Deleting consensus metadata
I20260809 06:50:39.705737 22809 catalog_manager.cc:4841] TS 435d70e8674c48ccbee7135f1349452c (127.27.209.195:46329): tablet 0c546334eb38460e8796865beaffda26 (table test-table [id=f8d485596a894c6e9f67d04e9aab5961]) successfully deleted
I20260809 06:50:39.707439 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.679272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.27.209.193:41405)
0809 06:50:39.679614 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.679632 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.688520 (+ 8888us) server_negotiation.cc:407] Connection header received
0809 06:50:39.689299 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.689306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.689416 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.689503 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:39.690232 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.691379 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.693245 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.693842 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.693937 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.706054 (+ 12117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:39.706064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:39.706078 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:39.706633 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:39.706691 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:39.706713 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:39.706816 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.706989 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.707104 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:50:39.708050 23121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:39.827960 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 22935
I20260809 06:50:39.850169 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 23135
I20260809 06:50:39.858011 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.851018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.27.209.194:41251)
0809 06:50:39.851200 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.851205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.851228 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:50:39.851424 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.851429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.851523 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.851682 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:39.852090 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.852829 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.854737 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.855176 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.855267 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.856638 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:39.856649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:39.856665 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:39.857281 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:39.857330 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:39.857351 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:39.857436 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.857577 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.857687 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:50:39.858712 23345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:39.946182 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 23135
I20260809 06:50:39.969453 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 23359
I20260809 06:50:39.976626 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:39.970494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.27.209.195:46329)
0809 06:50:39.970714 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:39.970720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:39.970743 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:50:39.970798 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:39.970802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:39.970888 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:39.970973 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:39.971409 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.972220 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.973502 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:39.973935 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:39.974061 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:39.975141 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:39.975151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:39.975167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:39.975876 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:39.975926 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:39.975951 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:39.976058 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:39.976189 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 06:50:39.976288 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:50:39.977355 23541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:40.099282 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 23359
I20260809 06:50:40.124550 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 22784
I20260809 06:50:40.125602 22886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:40.235596 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 22784
2026-08-09T06:50:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3026 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:50:40 dist-test-slave-jpj1 krb5kdc[23626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:50:40 dist-test-slave-jpj1 krb5kdc[23626](info): set up 2 sockets
Aug 09 06:50:40 dist-test-slave-jpj1 krb5kdc[23626](info): commencing operation
krb5kdc: starting...
W20260809 06:50:41.456189 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:50:41 dist-test-slave-jpj1 krb5kdc[23626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:50:41.724838 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:39247
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:42.055516 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:42.055824 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:42.055958 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:42.063320 23651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:42.063443 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:42.063526 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:42.063628 23651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:42.063692 23651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:42.074165 23651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:42.075469 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:42.076783 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:42.087884 23658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:42.087932 23659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:42.087972 23663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:42.089149 23651 server_base.cc:1034] running on GCE node
I20260809 06:50:42.089941 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:42.091288 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:42.092491 23651 hybrid_clock.cc:648] HybridClock initialized: now 1786258242092424 us; error 59 us; skew 500 ppm
I20260809 06:50:42.092855 23651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:42 dist-test-slave-jpj1 krb5kdc[23626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:42.095870 23651 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:50:42.097255 23651 webserver.cc:460] Webserver started at http://127.27.209.254:32839/ using document root <none> and password file <none>
I20260809 06:50:42.097918 23651 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:42.098037 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:42.098464 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:42.101557 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "1f1d2148caea403384dc4a4432c5ad52"
format_stamp: "Formatted at 2026-08-09 06:50:42 on dist-test-slave-jpj1"
server_key: "5256ff85cc78d57bc3b52639279de130"
server_key_iv: "2896bad77ddc52c8e6557406d557746e"
server_key_version: "encryptionkey@0"
I20260809 06:50:42.102311 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "1f1d2148caea403384dc4a4432c5ad52"
format_stamp: "Formatted at 2026-08-09 06:50:42 on dist-test-slave-jpj1"
server_key: "5256ff85cc78d57bc3b52639279de130"
server_key_iv: "2896bad77ddc52c8e6557406d557746e"
server_key_version: "encryptionkey@0"
I20260809 06:50:42.107482 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 06:50:42.110957 23679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:42.112324 23651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:42.112484 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "1f1d2148caea403384dc4a4432c5ad52"
format_stamp: "Formatted at 2026-08-09 06:50:42 on dist-test-slave-jpj1"
server_key: "5256ff85cc78d57bc3b52639279de130"
server_key_iv: "2896bad77ddc52c8e6557406d557746e"
server_key_version: "encryptionkey@0"
I20260809 06:50:42.112673 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:42.128139 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:42.131994 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:42.132246 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:42.149941 23651 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:39247
I20260809 06:50:42.149948 23755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:39247 every 8 connection(s)
I20260809 06:50:42.151281 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:42.155257 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:42.159639 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 23651
I20260809 06:50:42.159843 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:42.160401 28487 external_mini_cluster.cc:1428] Setting key 787cd5afe652ff51e99f0c130db7cb1a
I20260809 06:50:42.166769 23756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52: Bootstrap starting.
I20260809 06:50:42.170483 23756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:42.171742 23756 log.cc:824] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:42.175174 23756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52: No bootstrap required, opened a new log
Aug 09 06:50:42 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:42.184377 23756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d2148caea403384dc4a4432c5ad52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39247 } }
I20260809 06:50:42.184724 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:42.162320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.27.209.254:39247)
0809 06:50:42.163091 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:42.163102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:42.163152 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:50:42.165399 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:42.165425 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:42.165802 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:42.166124 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:42.168024 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:42.169375 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:42.171310 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:42.172105 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:42.172305 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:42.177264 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:42.177273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:42.177289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:42.177330 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:42.180551 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:42.181380 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:42.181386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:42.181393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:42.181562 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:42.181987 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:42.181990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:42.181992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:42.182537 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:42.182783 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:42.183249 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 06:50:42.183650 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
I20260809 06:50:42.184746 23756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:42.184978 23756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1d2148caea403384dc4a4432c5ad52, State: Initialized, Role: FOLLOWER
I20260809 06:50:42.185621 23756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d2148caea403384dc4a4432c5ad52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39247 } }
I20260809 06:50:42.185861 23756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:42.186003 23756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:42.186174 23756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:42.188872 23756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d2148caea403384dc4a4432c5ad52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39247 } }
I20260809 06:50:42.189317 23756 leader_election.cc:302] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f1d2148caea403384dc4a4432c5ad52; no voters:
I20260809 06:50:42.189862 23756 leader_election.cc:288] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:42.189994 23766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:42.190616 23766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1d2148caea403384dc4a4432c5ad52, State: Running, Role: LEADER
I20260809 06:50:42.191193 23766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d2148caea403384dc4a4432c5ad52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39247 } }
I20260809 06:50:42.191811 23756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:42.193995 23769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f1d2148caea403384dc4a4432c5ad52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d2148caea403384dc4a4432c5ad52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39247 } } }
I20260809 06:50:42.194211 23770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f1d2148caea403384dc4a4432c5ad52. Latest consensus state: current_term: 1 leader_uuid: "1f1d2148caea403384dc4a4432c5ad52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f1d2148caea403384dc4a4432c5ad52" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39247 } } }
I20260809 06:50:42.194401 23769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:42.194458 23770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:42.196714 23779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:42.201779 23779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:42.210785 23779 catalog_manager.cc:1347] Generated new cluster ID: c880a3c9bd5e4c778fa2ba5431db4632
I20260809 06:50:42.210902 23779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:42.230403 23779 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:42.232236 23779 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:42.244293 23779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52: Generated new TSK 0
I20260809 06:50:42.245198 23779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:50:42.318894 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:39247
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:42.654757 23795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:42.655026 23795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:42.655160 23795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:42.662225 23795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:42.662330 23795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:42.662478 23795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:42.671792 23795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:39247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:42.672978 23795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:42.674299 23795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:42.686590 23806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:42.686628 23807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:42.687321 23812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:42.687930 23795 server_base.cc:1034] running on GCE node
I20260809 06:50:42.688534 23795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:42.689419 23795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:42.690613 23795 hybrid_clock.cc:648] HybridClock initialized: now 1786258242690558 us; error 50 us; skew 500 ppm
I20260809 06:50:42.690899 23795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:42 dist-test-slave-jpj1 krb5kdc[23626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:42.694103 23795 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:50:42.695482 23795 webserver.cc:460] Webserver started at http://127.27.209.193:41729/ using document root <none> and password file <none>
I20260809 06:50:42.696189 23795 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:42.696336 23795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:42.696661 23795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:42.699502 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "c6ed27358a7c449ea82fd2269593ef5b"
format_stamp: "Formatted at 2026-08-09 06:50:42 on dist-test-slave-jpj1"
server_key: "e09e7c620bbb7bf2dd43bceb828fc81a"
server_key_iv: "898414d4b2ac89a218cf66bd6fd65ec7"
server_key_version: "encryptionkey@0"
I20260809 06:50:42.700274 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "c6ed27358a7c449ea82fd2269593ef5b"
format_stamp: "Formatted at 2026-08-09 06:50:42 on dist-test-slave-jpj1"
server_key: "e09e7c620bbb7bf2dd43bceb828fc81a"
server_key_iv: "898414d4b2ac89a218cf66bd6fd65ec7"
server_key_version: "encryptionkey@0"
I20260809 06:50:42.705194 23795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:50:42.708472 23822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:42.709775 23795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:50:42.709928 23795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "c6ed27358a7c449ea82fd2269593ef5b"
format_stamp: "Formatted at 2026-08-09 06:50:42 on dist-test-slave-jpj1"
server_key: "e09e7c620bbb7bf2dd43bceb828fc81a"
server_key_iv: "898414d4b2ac89a218cf66bd6fd65ec7"
server_key_version: "encryptionkey@0"
I20260809 06:50:42.710100 23795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:42.721728 23795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:42.725530 23795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:42.725793 23795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:42.726759 23795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:42.728092 23795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:42.728170 23795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:42.728296 23795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:42.728351 23795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:42.758723 23795 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:39671
I20260809 06:50:42.758797 23997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:39671 every 8 connection(s)
I20260809 06:50:42.760020 23795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:42.769133 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 23795
I20260809 06:50:42.769336 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:50:42.769886 28487 external_mini_cluster.cc:1428] Setting key cab45648219151d8f76996c1a8a5e230
Aug 09 06:50:42 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258242, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:42.780416 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:42.762497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:46447 (local address 127.27.209.254:39247)
0809 06:50:42.762933 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:42.762937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:42.763937 (+ 1000us) server_negotiation.cc:407] Connection header received
0809 06:50:42.765707 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:42.765712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:42.765798 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:42.765897 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:42.767388 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:42.768150 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:42.770231 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:42.770707 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:42.770839 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:42.774904 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:42.774909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:42.774912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:42.774944 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:42.777396 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:42.778208 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:42.778212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:42.778214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:42.778333 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:42.778768 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:42.778770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:42.778772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:42.779008 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:42.779117 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:42.779996 (+ 879us) server_negotiation.cc:299] Negotiation successful
0809 06:50:42.780167 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":211,"threads_started":1}
I20260809 06:50:42.781741 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:42.763038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:39247 (local address 127.27.209.193:46447)
0809 06:50:42.763770 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:42.763816 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:42.765439 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:42.766146 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:42.766154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:42.766690 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:42.767239 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:42.767258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:42.768306 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:42.768310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:42.770088 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:42.770110 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:42.770966 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:42.770974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:42.771189 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:42.771853 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:42.771883 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:42.774673 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:42.777532 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:42.777541 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:42.777558 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:42.778104 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:42.778425 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:42.778429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:42.778432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:42.778681 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:42.779129 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:42.779136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:42.779602 (+ 466us) client_negotiation.cc:769] Sending connection context
0809 06:50:42.779911 (+ 309us) client_negotiation.cc:240] Negotiation successful
0809 06:50:42.780419 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":264,"threads_started":1}
I20260809 06:50:42.783226 23999 heartbeater.cc:347] Connected to a master server at 127.27.209.254:39247
I20260809 06:50:42.783607 23999 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:42.784425 23999 heartbeater.cc:511] Master 127.27.209.254:39247 requested a full tablet report, sending...
I20260809 06:50:42.786607 23703 ts_manager.cc:194] Registered new tserver with Master: c6ed27358a7c449ea82fd2269593ef5b (127.27.209.193:39671)
I20260809 06:50:42.788100 23703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:46447
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:50:42.831280 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:39247
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:43.155812 24007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:43.156081 24007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:43.156219 24007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:43.163261 24007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:43.163364 24007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:43.163507 24007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:43.173187 24007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:39247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:43.174461 24007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:43.175827 24007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:43.188843 24014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:43.188817 24016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:43.188871 24020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:43.189782 24007 server_base.cc:1034] running on GCE node
I20260809 06:50:43.190397 24007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:43.191367 24007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:43.192567 24007 hybrid_clock.cc:648] HybridClock initialized: now 1786258243192493 us; error 63 us; skew 500 ppm
I20260809 06:50:43.192904 24007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:43 dist-test-slave-jpj1 krb5kdc[23626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:43.195988 24007 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:50:43.197461 24007 webserver.cc:460] Webserver started at http://127.27.209.194:37375/ using document root <none> and password file <none>
I20260809 06:50:43.198138 24007 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:43.198256 24007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:43.198550 24007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:43.201427 24007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "2fca917982804b8d9886c1c1486e3c6d"
format_stamp: "Formatted at 2026-08-09 06:50:43 on dist-test-slave-jpj1"
server_key: "660cd6c4d318a69982ae79ada25e1ec0"
server_key_iv: "b95757776be0e2fd3392ec3f8dccf43b"
server_key_version: "encryptionkey@0"
I20260809 06:50:43.202172 24007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "2fca917982804b8d9886c1c1486e3c6d"
format_stamp: "Formatted at 2026-08-09 06:50:43 on dist-test-slave-jpj1"
server_key: "660cd6c4d318a69982ae79ada25e1ec0"
server_key_iv: "b95757776be0e2fd3392ec3f8dccf43b"
server_key_version: "encryptionkey@0"
I20260809 06:50:43.207127 24007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:50:43.210528 24030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:43.211855 24007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:50:43.212008 24007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "2fca917982804b8d9886c1c1486e3c6d"
format_stamp: "Formatted at 2026-08-09 06:50:43 on dist-test-slave-jpj1"
server_key: "660cd6c4d318a69982ae79ada25e1ec0"
server_key_iv: "b95757776be0e2fd3392ec3f8dccf43b"
server_key_version: "encryptionkey@0"
I20260809 06:50:43.212198 24007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:43.234784 24007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:43.238901 24007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:43.239178 24007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:43.240199 24007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:43.241550 24007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:43.241632 24007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:43.241758 24007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:43.241812 24007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:43.272408 24007 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:42737
I20260809 06:50:43.272459 24193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:42737 every 8 connection(s)
I20260809 06:50:43.273712 24007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:43.276981 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 24007
I20260809 06:50:43.277199 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:50:43.277778 28487 external_mini_cluster.cc:1428] Setting key 4c26fceef9328cb3a8845387887434ea
Aug 09 06:50:43 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:43.294919 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.276188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:46091 (local address 127.27.209.254:39247)
0809 06:50:43.276366 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:43.276370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:43.277733 (+ 1363us) server_negotiation.cc:407] Connection header received
0809 06:50:43.279622 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:43.279628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:43.279720 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:43.279810 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:43.281316 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.282028 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.284144 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.284725 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.284836 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.288856 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:43.288862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:43.288866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:43.288903 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:43.291882 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:43.292654 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:43.292660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:43.292664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:43.292798 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:43.293238 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:43.293243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:43.293246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:43.293509 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:43.293659 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:43.294332 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 06:50:43.294558 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:50:43.295953 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.276716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:39247 (local address 127.27.209.194:46091)
0809 06:50:43.277557 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:43.277603 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:43.279346 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:43.280053 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:43.280062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:43.280578 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:43.281138 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.281160 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.282227 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.282233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.284006 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.284021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.285025 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.285032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.285304 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.286012 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:43.286039 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:43.288610 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:43.292005 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:43.292013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:43.292027 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:43.292537 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:43.292905 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:43.292910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:43.292913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:43.293138 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:43.293642 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:43.293650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:43.293982 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 06:50:43.294222 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 06:50:43.294738 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":350,"threads_started":1}
I20260809 06:50:43.297371 24195 heartbeater.cc:347] Connected to a master server at 127.27.209.254:39247
I20260809 06:50:43.297705 24195 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:43.298537 24195 heartbeater.cc:511] Master 127.27.209.254:39247 requested a full tablet report, sending...
I20260809 06:50:43.300263 23703 ts_manager.cc:194] Registered new tserver with Master: 2fca917982804b8d9886c1c1486e3c6d (127.27.209.194:42737)
I20260809 06:50:43.301311 23703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:46091
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:50:43.343170 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:39247
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:43.668262 24202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:43.668538 24202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:43.668644 24202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:43.675766 24202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:43.675873 24202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:43.675993 24202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:43.685487 24202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:39247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:43.686808 24202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:43.688117 24202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:43.700882 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:43.700846 24214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:43.700803 24212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:43.701656 24202 server_base.cc:1034] running on GCE node
I20260809 06:50:43.702208 24202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:43.703071 24202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:43.704269 24202 hybrid_clock.cc:648] HybridClock initialized: now 1786258243704209 us; error 44 us; skew 500 ppm
I20260809 06:50:43.704552 24202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:43 dist-test-slave-jpj1 krb5kdc[23626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:43.707434 24202 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:50:43.708815 24202 webserver.cc:460] Webserver started at http://127.27.209.195:40537/ using document root <none> and password file <none>
I20260809 06:50:43.709465 24202 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:43.709594 24202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:43.709930 24202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:43.712906 24202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "013685a948564a10bc70ff4250c6f413"
format_stamp: "Formatted at 2026-08-09 06:50:43 on dist-test-slave-jpj1"
server_key: "d1638eb6e2ea623ca701a267d3a1c6f3"
server_key_iv: "bef67cdd2ee25e147ab07404420a9b6d"
server_key_version: "encryptionkey@0"
I20260809 06:50:43.713685 24202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "013685a948564a10bc70ff4250c6f413"
format_stamp: "Formatted at 2026-08-09 06:50:43 on dist-test-slave-jpj1"
server_key: "d1638eb6e2ea623ca701a267d3a1c6f3"
server_key_iv: "bef67cdd2ee25e147ab07404420a9b6d"
server_key_version: "encryptionkey@0"
I20260809 06:50:43.718992 24202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:50:43.722267 24223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:43.723488 24202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:50:43.723675 24202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "013685a948564a10bc70ff4250c6f413"
format_stamp: "Formatted at 2026-08-09 06:50:43 on dist-test-slave-jpj1"
server_key: "d1638eb6e2ea623ca701a267d3a1c6f3"
server_key_iv: "bef67cdd2ee25e147ab07404420a9b6d"
server_key_version: "encryptionkey@0"
I20260809 06:50:43.723858 24202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:43.742242 24202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:43.745975 24202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:43.746259 24202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:43.747171 24202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:43.748530 24202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:43.748608 24202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:43.748728 24202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:43.748790 24202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:43.778465 24202 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:33143
I20260809 06:50:43.778504 24402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:33143 every 8 connection(s)
I20260809 06:50:43.779716 24202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:43.789242 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 24202
I20260809 06:50:43.789450 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:50:43.789944 28487 external_mini_cluster.cc:1428] Setting key fb49a49cc8c048168d2b884df98becd9
I20260809 06:50:43.790972 23999 heartbeater.cc:503] Master 127.27.209.254:39247 was elected leader, sending a full tablet report...
Aug 09 06:50:43 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258243, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:43.799139 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.782218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:39083 (local address 127.27.209.254:39247)
0809 06:50:43.782406 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:43.782411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:43.783470 (+ 1059us) server_negotiation.cc:407] Connection header received
0809 06:50:43.785410 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:43.785417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:43.785493 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:43.785586 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:43.787065 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.787875 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.789644 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.790078 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.790177 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.794062 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:43.794066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:43.794070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:43.794100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:43.796504 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:43.797155 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:43.797161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:43.797165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:43.797308 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:43.797685 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:43.797688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:43.797690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:43.797941 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:43.798050 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:43.798632 (+ 582us) server_negotiation.cc:299] Negotiation successful
0809 06:50:43.798864 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:50:43.800017 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.782625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:39247 (local address 127.27.209.195:39083)
0809 06:50:43.783319 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:43.783357 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:43.785125 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:43.785844 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:43.785852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:43.786355 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:43.786893 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.786912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.788031 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.788035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.789490 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.789506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.790323 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.790332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.790557 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.791256 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:43.791295 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:43.793872 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:43.796616 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:43.796622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:43.796633 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:43.797047 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:43.797416 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:43.797419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:43.797421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:43.797606 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:43.798030 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:43.798036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:43.798325 (+ 289us) client_negotiation.cc:769] Sending connection context
0809 06:50:43.798542 (+ 217us) client_negotiation.cc:240] Negotiation successful
0809 06:50:43.798932 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":266,"threads_started":1}
I20260809 06:50:43.801294 24403 heartbeater.cc:347] Connected to a master server at 127.27.209.254:39247
I20260809 06:50:43.801545 24403 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:43.802222 24403 heartbeater.cc:511] Master 127.27.209.254:39247 requested a full tablet report, sending...
I20260809 06:50:43.803500 23702 ts_manager.cc:194] Registered new tserver with Master: 013685a948564a10bc70ff4250c6f413 (127.27.209.195:33143)
I20260809 06:50:43.804500 23702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:39083
I20260809 06:50:43.805284 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:50:43.821985 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.809118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.27.209.254:39247)
0809 06:50:43.809279 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:43.809284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:43.809771 (+ 487us) server_negotiation.cc:407] Connection header received
0809 06:50:43.809891 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:43.809896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:43.809956 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:43.810042 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:43.811480 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.812219 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.813591 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.814057 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.814164 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.817019 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:43.817024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:43.817027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:43.817057 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:43.819307 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:43.820068 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:43.820073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:43.820075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:43.820202 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:43.820621 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:43.820625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:43.820627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:43.820916 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:43.821168 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:43.821530 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 06:50:43.821696 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:50:43.825568 23703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:50:43.832341 23703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:50:43.852895 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.843494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33143 (local address 127.0.0.1:53278)
0809 06:50:43.844618 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:43.844658 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:43.844835 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:43.845491 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:43.845499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:43.845536 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:43.846021 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.846035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.847432 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.847437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.849784 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.849799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.851839 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.851846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.852080 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.852135 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 06:50:43.852263 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 06:50:43.852432 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":23,"thread_start_us":831,"threads_started":1}
I20260809 06:50:43.853876 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.843700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.27.209.195:33143)
0809 06:50:43.844304 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:43.844314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:43.844768 (+ 454us) server_negotiation.cc:407] Connection header received
0809 06:50:43.845039 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:43.845047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:43.845288 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:43.845491 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:43.846248 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.847305 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.849966 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.851333 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.851508 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.851702 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:43.853346 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0809 06:50:43.853508 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":304,"threads_started":1}
I20260809 06:50:43.855705 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.844589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39671 (local address 127.0.0.1:56244)
0809 06:50:43.846140 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:43.846157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:43.846253 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:43.848329 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:43.848334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:43.848358 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:43.848798 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.848811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.850669 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.850674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.852693 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.852713 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.854931 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.854938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.855105 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.855137 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:50:43.855236 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 06:50:43.855354 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1413,"mutex_wait_us":1,"threads_started":1}
I20260809 06:50:43.856594 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.843976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.27.209.193:39671)
0809 06:50:43.847507 (+ 3531us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:43.847516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:43.847566 (+ 50us) server_negotiation.cc:407] Connection header received
0809 06:50:43.847665 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:43.847672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:43.847905 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:43.848086 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:43.848990 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.850507 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.852880 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.854406 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.854587 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.854781 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:43.856130 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0809 06:50:43.856295 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1150,"threads_started":1}
I20260809 06:50:43.858455 24301 tablet_service.cc:1467] Processing CreateTablet for tablet 0e65778913f542c09c73601269205c4c (DEFAULT_TABLE table=test-table [id=5233b28ad07248eeb8285465cae65c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:43.859918 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e65778913f542c09c73601269205c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:43.861104 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 0e65778913f542c09c73601269205c4c (DEFAULT_TABLE table=test-table [id=5233b28ad07248eeb8285465cae65c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:43.862610 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e65778913f542c09c73601269205c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:43.863759 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.845108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42737 (local address 127.0.0.1:59634)
0809 06:50:43.855878 (+ 10770us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:43.855895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:43.855982 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:43.857443 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:43.857448 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:43.857473 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:43.857663 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.857675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.859430 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.859436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.861215 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:43.861240 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.862969 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.862976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:43.863165 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.863196 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 06:50:43.863307 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 06:50:43.863435 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10686,"mutex_wait_us":3,"thread_start_us":754,"threads_started":1}
I20260809 06:50:43.863868 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:43.845581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59634 (local address 127.27.209.194:42737)
0809 06:50:43.856568 (+ 10987us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:43.856577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:43.856609 (+ 32us) server_negotiation.cc:407] Connection header received
0809 06:50:43.856705 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:43.856711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:43.856971 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:43.857176 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:43.857819 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.859285 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.861428 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:43.862804 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:43.862972 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:43.863130 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:43.863358 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 06:50:43.863519 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10818,"thread_start_us":11473,"threads_started":1}
I20260809 06:50:43.867506 24104 tablet_service.cc:1467] Processing CreateTablet for tablet 0e65778913f542c09c73601269205c4c (DEFAULT_TABLE table=test-table [id=5233b28ad07248eeb8285465cae65c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:43.869043 24104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e65778913f542c09c73601269205c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:43.871677 24432 tablet_bootstrap.cc:492] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Bootstrap starting.
I20260809 06:50:43.874585 24432 tablet_bootstrap.cc:654] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:43.875598 24432 log.cc:824] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:43.876464 24433 tablet_bootstrap.cc:492] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Bootstrap starting.
I20260809 06:50:43.880038 24433 tablet_bootstrap.cc:654] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:43.880806 24432 tablet_bootstrap.cc:492] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: No bootstrap required, opened a new log
I20260809 06:50:43.881090 24432 ts_tablet_manager.cc:1397] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260809 06:50:43.881296 24433 log.cc:824] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:43.884385 24433 tablet_bootstrap.cc:492] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: No bootstrap required, opened a new log
I20260809 06:50:43.884725 24433 ts_tablet_manager.cc:1397] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 06:50:43.885175 24436 tablet_bootstrap.cc:492] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Bootstrap starting.
I20260809 06:50:43.888139 24436 tablet_bootstrap.cc:654] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:43.889083 24436 log.cc:824] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:43.891428 24432 raft_consensus.cc:348] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:43.892040 24436 tablet_bootstrap.cc:492] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: No bootstrap required, opened a new log
I20260809 06:50:43.892014 24432 raft_consensus.cc:374] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:43.892117 24432 raft_consensus.cc:729] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013685a948564a10bc70ff4250c6f413, State: Initialized, Role: FOLLOWER
I20260809 06:50:43.892295 24436 ts_tablet_manager.cc:1397] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260809 06:50:43.892819 24432 consensus_queue.cc:260] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:43.893960 24403 heartbeater.cc:503] Master 127.27.209.254:39247 was elected leader, sending a full tablet report...
I20260809 06:50:43.894627 24432 ts_tablet_manager.cc:1428] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260809 06:50:43.894991 24433 raft_consensus.cc:348] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:43.895648 24433 raft_consensus.cc:374] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:43.895759 24433 raft_consensus.cc:729] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ed27358a7c449ea82fd2269593ef5b, State: Initialized, Role: FOLLOWER
I20260809 06:50:43.896350 24433 consensus_queue.cc:260] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:43.898100 24433 ts_tablet_manager.cc:1428] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260809 06:50:43.902621 24436 raft_consensus.cc:348] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:43.902985 24436 raft_consensus.cc:374] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:43.903072 24436 raft_consensus.cc:729] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fca917982804b8d9886c1c1486e3c6d, State: Initialized, Role: FOLLOWER
I20260809 06:50:43.903632 24436 consensus_queue.cc:260] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:43.904595 24195 heartbeater.cc:503] Master 127.27.209.254:39247 was elected leader, sending a full tablet report...
I20260809 06:50:43.905277 24436 ts_tablet_manager.cc:1428] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
W20260809 06:50:44.012338 24000 tablet.cc:2369] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:50:44.025568 24196 tablet.cc:2369] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 06:50:44.031241 24404 tablet.cc:2369] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:44.247481 24443 raft_consensus.cc:482] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:44.247792 24443 raft_consensus.cc:504] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:44.249444 24443 leader_election.cc:288] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 013685a948564a10bc70ff4250c6f413 (127.27.209.195:33143), 2fca917982804b8d9886c1c1486e3c6d (127.27.209.194:42737)
I20260809 06:50:44.255502 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.249901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:33679 (local address 127.27.209.195:33143)
0809 06:50:44.250070 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:44.250075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:44.250410 (+ 335us) server_negotiation.cc:407] Connection header received
0809 06:50:44.250653 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:44.250658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:44.250770 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:44.250863 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:44.251428 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.252170 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.253861 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.254625 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.254781 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.254858 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:44.255122 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 06:50:44.255249 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:50:44.255584 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.249681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:33143 (local address 127.27.209.193:33679)
0809 06:50:44.250310 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:44.250330 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:44.250476 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:44.250907 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:44.250912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:44.250940 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:44.251278 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:44.251288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.252296 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.252302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:44.253701 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:44.253713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.254820 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.254826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:44.254968 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.254992 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 06:50:44.255073 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:50:44.255190 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":309,"threads_started":1}
I20260809 06:50:44.255971 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.250663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:59827 (local address 127.27.209.194:42737)
0809 06:50:44.250880 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:44.250886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:44.251147 (+ 261us) server_negotiation.cc:407] Connection header received
0809 06:50:44.251323 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:44.251327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:44.251385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:44.251461 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:44.251994 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.252688 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.254333 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.255083 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.255220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.255300 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:44.255608 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 06:50:44.255733 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 06:50:44.256045 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.250418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42737 (local address 127.27.209.193:59827)
0809 06:50:44.251044 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:44.251063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:44.251188 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:44.251492 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:44.251497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:44.251520 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:44.251855 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:44.251864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.252839 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.252842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:44.254176 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:44.254188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.255304 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.255310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:44.255450 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.255473 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:50:44.255584 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 06:50:44.255714 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":465,"threads_started":1}
I20260809 06:50:44.256724 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e65778913f542c09c73601269205c4c" candidate_uuid: "c6ed27358a7c449ea82fd2269593ef5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013685a948564a10bc70ff4250c6f413" is_pre_election: true
I20260809 06:50:44.256871 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e65778913f542c09c73601269205c4c" candidate_uuid: "c6ed27358a7c449ea82fd2269593ef5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fca917982804b8d9886c1c1486e3c6d" is_pre_election: true
I20260809 06:50:44.257172 24333 raft_consensus.cc:2454] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ed27358a7c449ea82fd2269593ef5b in term 0.
I20260809 06:50:44.257349 24129 raft_consensus.cc:2454] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ed27358a7c449ea82fd2269593ef5b in term 0.
I20260809 06:50:44.257778 23826 leader_election.cc:302] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013685a948564a10bc70ff4250c6f413, c6ed27358a7c449ea82fd2269593ef5b; no voters:
I20260809 06:50:44.258134 24443 raft_consensus.cc:2781] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:50:44.258296 24443 raft_consensus.cc:482] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:50:44.258409 24443 raft_consensus.cc:3037] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:44.260658 24443 raft_consensus.cc:504] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:44.261250 24443 leader_election.cc:288] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [CANDIDATE]: Term 1 election: Requested vote from peers 013685a948564a10bc70ff4250c6f413 (127.27.209.195:33143), 2fca917982804b8d9886c1c1486e3c6d (127.27.209.194:42737)
I20260809 06:50:44.261672 24333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e65778913f542c09c73601269205c4c" candidate_uuid: "c6ed27358a7c449ea82fd2269593ef5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013685a948564a10bc70ff4250c6f413"
I20260809 06:50:44.261729 24129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e65778913f542c09c73601269205c4c" candidate_uuid: "c6ed27358a7c449ea82fd2269593ef5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fca917982804b8d9886c1c1486e3c6d"
I20260809 06:50:44.261852 24333 raft_consensus.cc:3037] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:44.261924 24129 raft_consensus.cc:3037] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:44.264148 24333 raft_consensus.cc:2454] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ed27358a7c449ea82fd2269593ef5b in term 1.
I20260809 06:50:44.264161 24129 raft_consensus.cc:2454] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ed27358a7c449ea82fd2269593ef5b in term 1.
I20260809 06:50:44.264505 23826 leader_election.cc:302] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fca917982804b8d9886c1c1486e3c6d, c6ed27358a7c449ea82fd2269593ef5b; no voters:
I20260809 06:50:44.264781 24443 raft_consensus.cc:2781] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:44.265189 24443 raft_consensus.cc:686] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 1 LEADER]: Becoming Leader. State: Replica: c6ed27358a7c449ea82fd2269593ef5b, State: Running, Role: LEADER
I20260809 06:50:44.265882 24443 consensus_queue.cc:237] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } }
I20260809 06:50:44.270205 23701 catalog_manager.cc:5495] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ed27358a7c449ea82fd2269593ef5b (127.27.209.193). New cstate: current_term: 1 leader_uuid: "c6ed27358a7c449ea82fd2269593ef5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ed27358a7c449ea82fd2269593ef5b" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 } health_report { overall_health: UNKNOWN } } }
I20260809 06:50:44.325776 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.320114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.27.209.193:39671)
0809 06:50:44.320327 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:44.320332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:44.320421 (+ 89us) server_negotiation.cc:407] Connection header received
0809 06:50:44.320668 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:44.320672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:44.320757 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:44.320892 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:50:44.321305 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.322027 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.323441 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.323955 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.324073 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.324671 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:50:44.325179 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:50:44.325340 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:44.325506 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 06:50:44.325554 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 06:50:44.334131 24333 raft_consensus.cc:1261] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 1 FOLLOWER]: Refusing update from remote peer c6ed27358a7c449ea82fd2269593ef5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:44.334142 24129 raft_consensus.cc:1261] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 1 FOLLOWER]: Refusing update from remote peer c6ed27358a7c449ea82fd2269593ef5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:44.335129 24449 consensus_queue.cc:1035] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fca917982804b8d9886c1c1486e3c6d" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:44.335479 24443 consensus_queue.cc:1035] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "013685a948564a10bc70ff4250c6f413" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:44.366900 23701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:44.367197 23701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:44.370865 23701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f1d2148caea403384dc4a4432c5ad52: Sending DeleteTablet for 3 replicas of tablet 0e65778913f542c09c73601269205c4c
I20260809 06:50:44.372085 23895 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e65778913f542c09c73601269205c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:44 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:56244
I20260809 06:50:44.372361 24104 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e65778913f542c09c73601269205c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:44 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:59634
I20260809 06:50:44.372393 24301 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e65778913f542c09c73601269205c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:44 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:53278
I20260809 06:50:44.373769 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 23795
I20260809 06:50:44.387296 24472 tablet_replica.cc:330] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: stopping tablet replica
I20260809 06:50:44.388425 24472 raft_consensus.cc:2229] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:44.389686 24472 raft_consensus.cc:2258] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:44.393118 24470 tablet_replica.cc:330] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: stopping tablet replica
I20260809 06:50:44.393518 24472 ts_tablet_manager.cc:1905] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:44.393666 24471 tablet_replica.cc:330] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: stopping tablet replica
I20260809 06:50:44.394207 24471 raft_consensus.cc:2229] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:44.394734 24470 raft_consensus.cc:2229] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:50:44.395319 24471 raft_consensus.cc:2258] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:44.396062 24470 raft_consensus.cc:2258] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:44.397991 24471 ts_tablet_manager.cc:1905] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:50:44 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:50:44.400709 24470 ts_tablet_manager.cc:1905] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:44.401612 24472 ts_tablet_manager.cc:1918] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:44.401873 24472 log.cc:1196] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/0e65778913f542c09c73601269205c4c
I20260809 06:50:44.402446 24472 ts_tablet_manager.cc:1939] T 0e65778913f542c09c73601269205c4c P 013685a948564a10bc70ff4250c6f413: Deleting consensus metadata
I20260809 06:50:44.404538 23683 catalog_manager.cc:4841] TS 013685a948564a10bc70ff4250c6f413 (127.27.209.195:33143): tablet 0e65778913f542c09c73601269205c4c (table test-table [id=5233b28ad07248eeb8285465cae65c69]) successfully deleted
I20260809 06:50:44.406168 24471 ts_tablet_manager.cc:1918] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:44.406339 24471 log.cc:1196] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/0e65778913f542c09c73601269205c4c
I20260809 06:50:44.406889 24471 ts_tablet_manager.cc:1939] T 0e65778913f542c09c73601269205c4c P 2fca917982804b8d9886c1c1486e3c6d: Deleting consensus metadata
I20260809 06:50:44.407136 24470 ts_tablet_manager.cc:1918] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:44.407297 24470 log.cc:1196] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/0e65778913f542c09c73601269205c4c
I20260809 06:50:44.407770 24470 ts_tablet_manager.cc:1939] T 0e65778913f542c09c73601269205c4c P c6ed27358a7c449ea82fd2269593ef5b: Deleting consensus metadata
I20260809 06:50:44.408823 23683 catalog_manager.cc:4841] TS 2fca917982804b8d9886c1c1486e3c6d (127.27.209.194:42737): tablet 0e65778913f542c09c73601269205c4c (table test-table [id=5233b28ad07248eeb8285465cae65c69]) successfully deleted
I20260809 06:50:44.409273 23680 catalog_manager.cc:4841] TS c6ed27358a7c449ea82fd2269593ef5b (127.27.209.193:39671): tablet 0e65778913f542c09c73601269205c4c (table test-table [id=5233b28ad07248eeb8285465cae65c69]) successfully deleted
I20260809 06:50:44.417037 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.375055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.27.209.193:39671)
0809 06:50:44.375231 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:44.375238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:44.386358 (+ 11120us) server_negotiation.cc:407] Connection header received
0809 06:50:44.386811 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:44.386818 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:44.386898 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:44.386968 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:44.388846 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.390097 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.393590 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.394271 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.394379 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.410446 (+ 16067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:44.410459 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:44.410478 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:44.410514 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:44.413583 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:44.414526 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:44.414534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:44.414536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:44.414696 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:44.415120 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:44.415123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:44.415125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:44.415488 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:44.415868 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:44.416236 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 06:50:44.416534 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:50:44.417717 23989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:44.514158 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 23795
I20260809 06:50:44.538355 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 24007
Aug 09 06:50:44 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:50:44.552590 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.539093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.27.209.194:42737)
0809 06:50:44.539285 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:44.539289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:44.539322 (+ 33us) server_negotiation.cc:407] Connection header received
0809 06:50:44.539481 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:44.539485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:44.539590 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:44.539677 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:44.540890 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.541623 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.542830 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.543233 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.543334 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.547364 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:44.547378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:44.547396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:44.547427 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:44.550033 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:44.550665 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:44.550672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:44.550674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:44.550811 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:44.551095 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:44.551098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:44.551100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:44.551481 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:44.551861 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:44.552078 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 06:50:44.552243 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:50:44.553256 24189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:44.643384 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 24007
I20260809 06:50:44.666724 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 24202
Aug 09 06:50:44 dist-test-slave-jpj1 krb5kdc[23626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:50:44.680977 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:44.667444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.27.209.195:33143)
0809 06:50:44.667660 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:44.667666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:44.667686 (+ 20us) server_negotiation.cc:407] Connection header received
0809 06:50:44.667809 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:44.667812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:44.667878 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:44.667969 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:44.669199 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.669977 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.671154 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:44.671523 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:44.671806 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:44.676055 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:44.676072 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:44.676090 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:44.676123 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:44.678584 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:44.679208 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:44.679215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:44.679217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:44.679345 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:44.679668 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:44.679671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:44.679673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:44.680031 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:44.680252 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:44.680472 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 06:50:44.680639 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:50:44.681624 24395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:44.767441 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 24202
I20260809 06:50:44.789737 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 23651
I20260809 06:50:44.790844 23749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:44.899616 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 23651
2026-08-09T06:50:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T06:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:44Z Disabled control of system clock
I20260809 06:50:44.986506 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44707
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:50:45.320114 24489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:45.320398 24489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:45.320487 24489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:45.327694 24489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:45.327823 24489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:45.327867 24489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:45.327934 24489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:45.327981 24489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:45.337307 24489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:45.338616 24489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:45.340479 24489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:45.352084 24498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:45.352123 24496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:45.352043 24500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:45.352787 24489 server_base.cc:1034] running on GCE node
I20260809 06:50:45.353541 24489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:45.354982 24489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:45.356220 24489 hybrid_clock.cc:648] HybridClock initialized: now 1786258245356159 us; error 49 us; skew 500 ppm
I20260809 06:50:45.356570 24489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:45.362680 24489 webserver.cc:460] Webserver started at http://127.27.209.254:42061/ using document root <none> and password file <none>
I20260809 06:50:45.363337 24489 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:45.363425 24489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:45.363737 24489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:45.366565 24489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "9240d82f752542198b5ceddb827cd473"
format_stamp: "Formatted at 2026-08-09 06:50:45 on dist-test-slave-jpj1"
server_key: "6f55b49b2b87eb8b8817802c5e1ba7d8"
server_key_iv: "240402567be6c2f58340fbe2f21418c7"
server_key_version: "encryptionkey@0"
I20260809 06:50:45.367321 24489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "9240d82f752542198b5ceddb827cd473"
format_stamp: "Formatted at 2026-08-09 06:50:45 on dist-test-slave-jpj1"
server_key: "6f55b49b2b87eb8b8817802c5e1ba7d8"
server_key_iv: "240402567be6c2f58340fbe2f21418c7"
server_key_version: "encryptionkey@0"
I20260809 06:50:45.372371 24489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 06:50:45.375604 24507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:45.376845 24489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:50:45.376986 24489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "9240d82f752542198b5ceddb827cd473"
format_stamp: "Formatted at 2026-08-09 06:50:45 on dist-test-slave-jpj1"
server_key: "6f55b49b2b87eb8b8817802c5e1ba7d8"
server_key_iv: "240402567be6c2f58340fbe2f21418c7"
server_key_version: "encryptionkey@0"
I20260809 06:50:45.377153 24489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:45.389833 24489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:45.390559 24489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:45.390791 24489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:45.410872 24489 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:44707
I20260809 06:50:45.410909 24600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:44707 every 8 connection(s)
I20260809 06:50:45.412631 24489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:45.416882 24601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:45.419211 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 24489
I20260809 06:50:45.419379 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:45.419962 28487 external_mini_cluster.cc:1428] Setting key 457f9eb101adc1a1a23daa0674318df2
I20260809 06:50:45.432432 24601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473: Bootstrap starting.
I20260809 06:50:45.436344 24601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:45.437767 24601 log.cc:824] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:45.441331 24601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473: No bootstrap required, opened a new log
I20260809 06:50:45.447337 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:45.421987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.27.209.254:44707)
0809 06:50:45.432419 (+ 10432us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:45.432434 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:45.432491 (+ 57us) server_negotiation.cc:407] Connection header received
0809 06:50:45.434910 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:45.434942 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:45.435595 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:45.436021 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:45.437305 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:45.438891 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:45.440947 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:45.441969 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:45.442707 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:45.444249 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:45.444258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:45.444283 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:45.444901 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:45.444994 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:45.445016 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:45.445168 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:45.445672 (+ 504us) server_negotiation.cc:299] Negotiation successful
0809 06:50:45.446003 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9857,"thread_start_us":9589,"threads_started":1}
I20260809 06:50:45.451484 24601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9240d82f752542198b5ceddb827cd473" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44707 } }
I20260809 06:50:45.451881 24601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:45.451984 24601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9240d82f752542198b5ceddb827cd473, State: Initialized, Role: FOLLOWER
I20260809 06:50:45.452548 24601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9240d82f752542198b5ceddb827cd473" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44707 } }
I20260809 06:50:45.452782 24601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:45.452909 24601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:45.453063 24601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:45.455124 24601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9240d82f752542198b5ceddb827cd473" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44707 } }
I20260809 06:50:45.455634 24601 leader_election.cc:302] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9240d82f752542198b5ceddb827cd473; no voters:
I20260809 06:50:45.456239 24601 leader_election.cc:288] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:45.456401 24611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:45.457116 24611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [term 1 LEADER]: Becoming Leader. State: Replica: 9240d82f752542198b5ceddb827cd473, State: Running, Role: LEADER
I20260809 06:50:45.457605 24611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9240d82f752542198b5ceddb827cd473" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44707 } }
I20260809 06:50:45.458195 24601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:45.460232 24613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9240d82f752542198b5ceddb827cd473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9240d82f752542198b5ceddb827cd473" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44707 } } }
I20260809 06:50:45.460542 24613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:45.461036 24614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9240d82f752542198b5ceddb827cd473. Latest consensus state: current_term: 1 leader_uuid: "9240d82f752542198b5ceddb827cd473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9240d82f752542198b5ceddb827cd473" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44707 } } }
I20260809 06:50:45.461346 24614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:45.465045 24618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:45.469813 24618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:45.478356 24618 catalog_manager.cc:1347] Generated new cluster ID: 5ce8be21e8ce41cc93916151864b07b0
I20260809 06:50:45.478475 24618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:45.499467 24618 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:45.500771 24618 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:45.506815 24618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473: Generated new TSK 0
I20260809 06:50:45.507658 24618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:50:45.517508 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44707
--builtin_ntp_servers=127.27.209.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:50:45.861999 24638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:45.862275 24638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:45.862392 24638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:45.869436 24638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:45.869536 24638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:45.869695 24638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:45.878576 24638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:45.879881 24638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:45.881754 24638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:45.894289 24646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:45.894413 24643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:45.894424 24644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:45.895715 24638 server_base.cc:1034] running on GCE node
I20260809 06:50:45.896261 24638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:45.897111 24638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:45.898309 24638 hybrid_clock.cc:648] HybridClock initialized: now 1786258245898253 us; error 40 us; skew 500 ppm
I20260809 06:50:45.898590 24638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:45.900661 24638 webserver.cc:460] Webserver started at http://127.27.209.193:33237/ using document root <none> and password file <none>
I20260809 06:50:45.901309 24638 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:45.901410 24638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:45.901710 24638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:45.904565 24638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "883aa6c75aba46a9b261cae6f6bd25db"
format_stamp: "Formatted at 2026-08-09 06:50:45 on dist-test-slave-jpj1"
server_key: "339e0d836bd83e311d125cdf1e2dadf4"
server_key_iv: "835baad88938be5e82c94a2f47540c98"
server_key_version: "encryptionkey@0"
I20260809 06:50:45.905333 24638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "883aa6c75aba46a9b261cae6f6bd25db"
format_stamp: "Formatted at 2026-08-09 06:50:45 on dist-test-slave-jpj1"
server_key: "339e0d836bd83e311d125cdf1e2dadf4"
server_key_iv: "835baad88938be5e82c94a2f47540c98"
server_key_version: "encryptionkey@0"
I20260809 06:50:45.910221 24638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:50:45.913450 24652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:45.914592 24638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 06:50:45.914747 24638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "883aa6c75aba46a9b261cae6f6bd25db"
format_stamp: "Formatted at 2026-08-09 06:50:45 on dist-test-slave-jpj1"
server_key: "339e0d836bd83e311d125cdf1e2dadf4"
server_key_iv: "835baad88938be5e82c94a2f47540c98"
server_key_version: "encryptionkey@0"
I20260809 06:50:45.914937 24638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:45.928399 24638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:45.929170 24638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:45.929464 24638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:45.930405 24638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:45.931843 24638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:45.931942 24638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:45.932045 24638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:45.932116 24638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:45.962807 24638 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:38519
I20260809 06:50:45.962857 24828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:38519 every 8 connection(s)
I20260809 06:50:45.964274 24638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:45.969615 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 24638
I20260809 06:50:45.969790 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:50:45.970220 28487 external_mini_cluster.cc:1428] Setting key 19b427a941f2141b373876f5340787de
I20260809 06:50:45.973073 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44707
--builtin_ntp_servers=127.27.209.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:50:45.976887 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:45.966819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:53189 (local address 127.27.209.254:44707)
0809 06:50:45.967213 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:45.967218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:45.968461 (+ 1243us) server_negotiation.cc:407] Connection header received
0809 06:50:45.970273 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:45.970280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:45.970365 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:45.970487 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:45.971621 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:45.972527 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:45.974114 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:45.974696 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:45.974821 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:45.976116 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:45.976122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:45.976126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:45.976146 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:45.976195 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:45.976210 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:45.976303 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:45.976470 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 06:50:45.976571 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":194,"threads_started":1}
I20260809 06:50:45.977752 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:45.967409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44707 (local address 127.27.209.193:53189)
0809 06:50:45.968316 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:45.968353 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:45.970000 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:45.970668 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:45.970675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:45.970765 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:45.971428 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:45.971452 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:45.972673 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:45.972677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:45.973969 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:45.973982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:45.974884 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:45.974889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:45.975140 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:45.975808 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:45.975810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:45.975849 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:45.975932 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:45.976298 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:45.976304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:45.976309 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:50:45.976431 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 06:50:45.976731 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":281,"threads_started":1}
I20260809 06:50:45.979087 24830 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44707
I20260809 06:50:45.979410 24830 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:45.980311 24830 heartbeater.cc:511] Master 127.27.209.254:44707 requested a full tablet report, sending...
I20260809 06:50:45.982393 24533 ts_manager.cc:194] Registered new tserver with Master: 883aa6c75aba46a9b261cae6f6bd25db (127.27.209.193:38519)
I20260809 06:50:45.983891 24533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.193:53189
W20260809 06:50:46.310575 24837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:46.310871 24837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:46.310959 24837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:46.318069 24837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:46.318172 24837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:46.318290 24837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:46.327634 24837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:46.328897 24837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:46.331015 24837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:46.343854 24844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:46.343971 24843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:46.343969 24849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:46.344825 24837 server_base.cc:1034] running on GCE node
I20260809 06:50:46.345420 24837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:46.346320 24837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:46.347507 24837 hybrid_clock.cc:648] HybridClock initialized: now 1786258246347457 us; error 49 us; skew 500 ppm
I20260809 06:50:46.347817 24837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:46.349994 24837 webserver.cc:460] Webserver started at http://127.27.209.194:40371/ using document root <none> and password file <none>
I20260809 06:50:46.350654 24837 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:46.350740 24837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:46.350997 24837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:46.353881 24837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "88f8f548e6a14492b229470f769fa2de"
format_stamp: "Formatted at 2026-08-09 06:50:46 on dist-test-slave-jpj1"
server_key: "0dea221431dad9b426412d14591e2964"
server_key_iv: "62ea4e5726c5082ce24cdbba8d03a373"
server_key_version: "encryptionkey@0"
I20260809 06:50:46.354619 24837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "88f8f548e6a14492b229470f769fa2de"
format_stamp: "Formatted at 2026-08-09 06:50:46 on dist-test-slave-jpj1"
server_key: "0dea221431dad9b426412d14591e2964"
server_key_iv: "62ea4e5726c5082ce24cdbba8d03a373"
server_key_version: "encryptionkey@0"
I20260809 06:50:46.359578 24837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 06:50:46.363023 24858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:46.364332 24837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:50:46.364468 24837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "88f8f548e6a14492b229470f769fa2de"
format_stamp: "Formatted at 2026-08-09 06:50:46 on dist-test-slave-jpj1"
server_key: "0dea221431dad9b426412d14591e2964"
server_key_iv: "62ea4e5726c5082ce24cdbba8d03a373"
server_key_version: "encryptionkey@0"
I20260809 06:50:46.364634 24837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:46.394776 24837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:46.395629 24837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:46.395924 24837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:46.396878 24837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:46.398329 24837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:46.398428 24837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:46.398548 24837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:46.398619 24837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:46.430279 24837 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:34753
I20260809 06:50:46.430348 25034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:34753 every 8 connection(s)
I20260809 06:50:46.431505 24837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:46.432790 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 24837
I20260809 06:50:46.432969 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:50:46.433387 28487 external_mini_cluster.cc:1428] Setting key 27c0083e1bf0f39e0c6b073e7334034e
I20260809 06:50:46.436242 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44707
--builtin_ntp_servers=127.27.209.212:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:50:46.443647 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.434081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:42809 (local address 127.27.209.254:44707)
0809 06:50:46.434260 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.434264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.435856 (+ 1592us) server_negotiation.cc:407] Connection header received
0809 06:50:46.437651 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.437657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.437761 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.437846 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:46.438903 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.439655 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.441147 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.441521 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.441630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.442889 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:46.442895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:46.442898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:46.442915 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:46.442946 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:46.442957 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:46.443030 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:46.443238 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 06:50:46.443332 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 06:50:46.444614 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.434650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44707 (local address 127.27.209.194:42809)
0809 06:50:46.435695 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.435745 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.437399 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.438089 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.438096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.438181 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:46.438741 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.438758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.439804 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.439808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.441007 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.441019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.441732 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.441740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.441954 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.442588 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:46.442591 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:46.442630 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:46.442703 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:46.443043 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:46.443054 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:46.443061 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 06:50:46.443211 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 06:50:46.443616 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":311,"threads_started":1}
I20260809 06:50:46.445982 25036 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44707
I20260809 06:50:46.446262 25036 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:46.447018 25036 heartbeater.cc:511] Master 127.27.209.254:44707 requested a full tablet report, sending...
I20260809 06:50:46.448437 24533 ts_manager.cc:194] Registered new tserver with Master: 88f8f548e6a14492b229470f769fa2de (127.27.209.194:34753)
I20260809 06:50:46.449213 24533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.194:42809
W20260809 06:50:46.761439 25043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:46.761711 25043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:46.761847 25043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:46.768980 25043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:46.769076 25043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:46.769207 25043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:46.778170 25043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:46.779405 25043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:46.781283 25043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:46.795053 25055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:46.795161 25052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:46.796293 25057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:46.797089 25043 server_base.cc:1034] running on GCE node
I20260809 06:50:46.797712 25043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:46.798763 25043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:46.799979 25043 hybrid_clock.cc:648] HybridClock initialized: now 1786258246799907 us; error 55 us; skew 500 ppm
I20260809 06:50:46.800367 25043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:46.803010 25043 webserver.cc:460] Webserver started at http://127.27.209.195:35735/ using document root <none> and password file <none>
I20260809 06:50:46.803785 25043 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:46.803890 25043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:46.804196 25043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:46.807054 25043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "8bd358631fec461fb489fac1d07e1dda"
format_stamp: "Formatted at 2026-08-09 06:50:46 on dist-test-slave-jpj1"
server_key: "247afedf8a15e9cd253a7a78f2ddeada"
server_key_iv: "c8586d81b800b85fb54293ce90935276"
server_key_version: "encryptionkey@0"
I20260809 06:50:46.807866 25043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "8bd358631fec461fb489fac1d07e1dda"
format_stamp: "Formatted at 2026-08-09 06:50:46 on dist-test-slave-jpj1"
server_key: "247afedf8a15e9cd253a7a78f2ddeada"
server_key_iv: "c8586d81b800b85fb54293ce90935276"
server_key_version: "encryptionkey@0"
I20260809 06:50:46.812700 25043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 06:50:46.815955 25068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:46.817188 25043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:50:46.817348 25043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "8bd358631fec461fb489fac1d07e1dda"
format_stamp: "Formatted at 2026-08-09 06:50:46 on dist-test-slave-jpj1"
server_key: "247afedf8a15e9cd253a7a78f2ddeada"
server_key_iv: "c8586d81b800b85fb54293ce90935276"
server_key_version: "encryptionkey@0"
I20260809 06:50:46.817535 25043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:46.832903 25043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:46.833613 25043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:46.833866 25043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:46.834692 25043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:46.836071 25043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:46.836164 25043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:46.836241 25043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:46.836293 25043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:46.865672 25043 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:40197
I20260809 06:50:46.865724 25256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:40197 every 8 connection(s)
I20260809 06:50:46.866945 25043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:46.873847 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 25043
I20260809 06:50:46.874006 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:50:46.874504 28487 external_mini_cluster.cc:1428] Setting key 0e50d4f5a03fc3e70f105052d8f7c0f0
I20260809 06:50:46.878894 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.869414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:48453 (local address 127.27.209.254:44707)
0809 06:50:46.869639 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.869644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.870939 (+ 1295us) server_negotiation.cc:407] Connection header received
0809 06:50:46.872613 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.872617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.872682 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.872770 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:46.873777 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.874584 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.876520 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.876951 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.877079 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.878271 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:46.878276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:46.878280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:46.878296 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:46.878328 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:46.878339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:46.878407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:46.878562 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 06:50:46.878645 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 06:50:46.879881 25263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.869922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44707 (local address 127.27.209.195:48453)
0809 06:50:46.870807 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.870849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.872385 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.872965 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.872973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.873055 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:46.873588 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.873604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.874725 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.874731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.876385 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.876401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.877181 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.877189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.877419 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.877992 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:46.877995 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:46.878031 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:46.878104 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:46.878412 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:46.878418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:46.878423 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 06:50:46.878550 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 06:50:46.878851 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":265,"threads_started":1}
I20260809 06:50:46.881129 25260 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44707
I20260809 06:50:46.881392 25260 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:46.882136 25260 heartbeater.cc:511] Master 127.27.209.254:44707 requested a full tablet report, sending...
I20260809 06:50:46.883450 24533 ts_manager.cc:194] Registered new tserver with Master: 8bd358631fec461fb489fac1d07e1dda (127.27.209.195:40197)
I20260809 06:50:46.884227 24533 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.195:48453
I20260809 06:50:46.889147 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:50:46.898638 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.892932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.27.209.254:44707)
0809 06:50:46.893102 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.893109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.893509 (+ 400us) server_negotiation.cc:407] Connection header received
0809 06:50:46.893676 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.893679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.893756 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.893835 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:46.894315 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.894993 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.896384 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.896787 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.896896 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.897960 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:46.897965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:46.897968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:46.897985 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:46.898024 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:46.898035 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:46.898155 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:46.898348 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 06:50:46.898405 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 06:50:46.902217 24533 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:50:46.908869 24533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:50:46.940753 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.919931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:34753 (local address 127.0.0.1:38680)
0809 06:50:46.920645 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.920693 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.920881 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.933798 (+ 12917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.933806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.933836 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:46.934285 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.934300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.936053 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.936059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.938282 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.938299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.939813 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.939819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.940021 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.940074 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 06:50:46.940190 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 06:50:46.940389 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":2,"thread_start_us":298,"threads_started":1}
I20260809 06:50:46.940788 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.920309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.27.209.194:34753)
0809 06:50:46.933087 (+ 12778us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.933098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.933135 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:50:46.933262 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.933271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.933555 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.933779 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:46.934499 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.935930 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.938478 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.939614 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.939804 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.940044 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:46.940269 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 06:50:46.940427 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12568,"thread_start_us":13696,"threads_started":1}
I20260809 06:50:46.942605 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.921067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:40197 (local address 127.0.0.1:35466)
0809 06:50:46.927890 (+ 6823us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.927911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.928046 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.935236 (+ 7190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.935248 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.935273 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:46.935665 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.935678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.937360 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.937364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.939776 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.939795 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.941871 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.941878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.942051 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.942085 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:50:46.942153 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 06:50:46.942281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":105,"thread_start_us":269,"threads_started":1}
I20260809 06:50:46.942989 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.921710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38519 (local address 127.0.0.1:49108)
0809 06:50:46.932895 (+ 11185us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.932915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.933029 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.935558 (+ 2529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.935563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.935594 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:46.936106 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.936121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.937883 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.937888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.940092 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.940105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.942196 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.942203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:46.942373 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.942406 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 06:50:46.942493 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 06:50:46.942623 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11062,"mutex_wait_us":68,"thread_start_us":11434,"threads_started":1}
I20260809 06:50:46.943030 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.921945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.27.209.193:38519)
0809 06:50:46.934936 (+ 12991us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.934944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.934969 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:50:46.935052 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.935059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.935259 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.935434 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:46.936288 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.937727 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.940307 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.941995 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.942149 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.942354 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:46.942566 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 06:50:46.942733 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12845,"thread_start_us":12727,"threads_started":1}
I20260809 06:50:46.943818 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.921427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.27.209.195:40197)
0809 06:50:46.934664 (+ 13237us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.934673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.934710 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:50:46.934815 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.934822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.935036 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.935240 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:46.935878 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.937198 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.939990 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.941341 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.941521 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:46.941711 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:46.943296 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0809 06:50:46.943460 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13065,"thread_start_us":12929,"threads_started":1}
I20260809 06:50:46.945511 24934 tablet_service.cc:1467] Processing CreateTablet for tablet 8333fc768c884b0e92a9a0ac6d120763 (DEFAULT_TABLE table=test-table [id=2a5f026fbe67492480ad4018906b41d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:46.947099 24934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8333fc768c884b0e92a9a0ac6d120763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:46.946700 24720 tablet_service.cc:1467] Processing CreateTablet for tablet 8333fc768c884b0e92a9a0ac6d120763 (DEFAULT_TABLE table=test-table [id=2a5f026fbe67492480ad4018906b41d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:46.946942 25139 tablet_service.cc:1467] Processing CreateTablet for tablet 8333fc768c884b0e92a9a0ac6d120763 (DEFAULT_TABLE table=test-table [id=2a5f026fbe67492480ad4018906b41d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:46.948088 24720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8333fc768c884b0e92a9a0ac6d120763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:46.948145 25139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8333fc768c884b0e92a9a0ac6d120763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:46.959326 25295 tablet_bootstrap.cc:492] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Bootstrap starting.
I20260809 06:50:46.961874 25298 tablet_bootstrap.cc:492] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Bootstrap starting.
I20260809 06:50:46.963182 25295 tablet_bootstrap.cc:654] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:46.964224 25296 tablet_bootstrap.cc:492] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Bootstrap starting.
I20260809 06:50:46.964476 25295 log.cc:824] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:46.967512 25295 tablet_bootstrap.cc:492] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: No bootstrap required, opened a new log
I20260809 06:50:46.967871 25295 ts_tablet_manager.cc:1397] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260809 06:50:46.969096 25298 tablet_bootstrap.cc:654] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:46.969552 25296 tablet_bootstrap.cc:654] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:46.970474 25298 log.cc:824] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:46.970844 25296 log.cc:824] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:46.977864 25296 tablet_bootstrap.cc:492] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: No bootstrap required, opened a new log
I20260809 06:50:46.978044 25298 tablet_bootstrap.cc:492] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: No bootstrap required, opened a new log
I20260809 06:50:46.978124 25296 ts_tablet_manager.cc:1397] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260809 06:50:46.978308 25298 ts_tablet_manager.cc:1397] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260809 06:50:46.977991 25295 raft_consensus.cc:348] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.978467 25295 raft_consensus.cc:374] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:46.978569 25295 raft_consensus.cc:729] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd358631fec461fb489fac1d07e1dda, State: Initialized, Role: FOLLOWER
I20260809 06:50:46.979290 25295 consensus_queue.cc:260] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.980542 25260 heartbeater.cc:503] Master 127.27.209.254:44707 was elected leader, sending a full tablet report...
I20260809 06:50:46.981146 25295 ts_tablet_manager.cc:1428] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260809 06:50:46.986552 24830 heartbeater.cc:503] Master 127.27.209.254:44707 was elected leader, sending a full tablet report...
I20260809 06:50:46.987692 25298 raft_consensus.cc:348] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.988139 25298 raft_consensus.cc:374] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:46.987866 25296 raft_consensus.cc:348] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.988238 25298 raft_consensus.cc:729] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883aa6c75aba46a9b261cae6f6bd25db, State: Initialized, Role: FOLLOWER
I20260809 06:50:46.988292 25296 raft_consensus.cc:374] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:46.988387 25296 raft_consensus.cc:729] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f8f548e6a14492b229470f769fa2de, State: Initialized, Role: FOLLOWER
I20260809 06:50:46.988842 25298 consensus_queue.cc:260] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.988930 25296 consensus_queue.cc:260] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.990644 25298 ts_tablet_manager.cc:1428] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 06:50:46.991627 25036 heartbeater.cc:503] Master 127.27.209.254:44707 was elected leader, sending a full tablet report...
I20260809 06:50:46.992638 25313 raft_consensus.cc:482] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:46.992903 25313 raft_consensus.cc:504] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:46.995635 25296 ts_tablet_manager.cc:1428] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Time spent starting tablet: real 0.017s user 0.004s sys 0.015s
I20260809 06:50:46.997278 25313 leader_election.cc:288] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bd358631fec461fb489fac1d07e1dda (127.27.209.195:40197), 883aa6c75aba46a9b261cae6f6bd25db (127.27.209.193:38519)
I20260809 06:50:47.002767 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.996008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:40893 (local address 127.27.209.193:38519)
0809 06:50:46.996393 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.996400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.997508 (+ 1108us) server_negotiation.cc:407] Connection header received
0809 06:50:46.997570 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.997574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.997640 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.997722 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:46.998378 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.999102 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.001145 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.001887 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.002027 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.002113 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:47.002401 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 06:50:47.002524 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260809 06:50:47.002797 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.995775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:46943 (local address 127.27.209.195:40197)
0809 06:50:46.995944 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:46.995950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:46.996861 (+ 911us) server_negotiation.cc:407] Connection header received
0809 06:50:46.997635 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:46.997640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:46.997695 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:46.997775 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:46.998444 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.999147 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.001116 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.001889 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.002043 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.002113 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:47.002440 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 06:50:47.002560 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:50:47.002880 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.995319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:40197 (local address 127.27.209.194:46943)
0809 06:50:46.996754 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.996775 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.997505 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.997801 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.997805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.997828 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:46.998301 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.998310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.999287 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.999291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:47.000922 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:47.000935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.002128 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.002137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:47.002298 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.002321 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 06:50:47.002401 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 06:50:47.002524 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":95,"thread_start_us":1110,"threads_started":1}
I20260809 06:50:47.003196 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:46.995576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38519 (local address 127.27.209.194:40893)
0809 06:50:46.997271 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:46.997287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:46.997421 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:46.997760 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:46.997765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:46.997790 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:46.998197 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:46.998210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:46.999223 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:46.999228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:47.000924 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:47.000937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.002082 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.002090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:47.002239 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.002281 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 06:50:47.002361 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 06:50:47.002498 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"mutex_wait_us":78,"thread_start_us":1439,"threads_started":1}
I20260809 06:50:47.003762 25176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8333fc768c884b0e92a9a0ac6d120763" candidate_uuid: "88f8f548e6a14492b229470f769fa2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd358631fec461fb489fac1d07e1dda" is_pre_election: true
I20260809 06:50:47.004048 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8333fc768c884b0e92a9a0ac6d120763" candidate_uuid: "88f8f548e6a14492b229470f769fa2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883aa6c75aba46a9b261cae6f6bd25db" is_pre_election: true
I20260809 06:50:47.004258 25176 raft_consensus.cc:2454] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f8f548e6a14492b229470f769fa2de in term 0.
I20260809 06:50:47.004513 24753 raft_consensus.cc:2454] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88f8f548e6a14492b229470f769fa2de in term 0.
I20260809 06:50:47.004865 24860 leader_election.cc:302] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88f8f548e6a14492b229470f769fa2de, 8bd358631fec461fb489fac1d07e1dda; no voters:
I20260809 06:50:47.005232 25313 raft_consensus.cc:2781] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:50:47.005373 25313 raft_consensus.cc:482] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:50:47.005465 25313 raft_consensus.cc:3037] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:47.007668 25313 raft_consensus.cc:504] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:47.008301 25313 leader_election.cc:288] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [CANDIDATE]: Term 1 election: Requested vote from peers 8bd358631fec461fb489fac1d07e1dda (127.27.209.195:40197), 883aa6c75aba46a9b261cae6f6bd25db (127.27.209.193:38519)
I20260809 06:50:47.008730 25176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8333fc768c884b0e92a9a0ac6d120763" candidate_uuid: "88f8f548e6a14492b229470f769fa2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd358631fec461fb489fac1d07e1dda"
I20260809 06:50:47.008777 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8333fc768c884b0e92a9a0ac6d120763" candidate_uuid: "88f8f548e6a14492b229470f769fa2de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883aa6c75aba46a9b261cae6f6bd25db"
I20260809 06:50:47.008955 24753 raft_consensus.cc:3037] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:47.008944 25176 raft_consensus.cc:3037] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:47.011384 25176 raft_consensus.cc:2454] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f8f548e6a14492b229470f769fa2de in term 1.
I20260809 06:50:47.011394 24753 raft_consensus.cc:2454] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f8f548e6a14492b229470f769fa2de in term 1.
I20260809 06:50:47.011780 24859 leader_election.cc:302] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 883aa6c75aba46a9b261cae6f6bd25db, 88f8f548e6a14492b229470f769fa2de; no voters:
I20260809 06:50:47.012085 25313 raft_consensus.cc:2781] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:47.012231 25313 raft_consensus.cc:686] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 1 LEADER]: Becoming Leader. State: Replica: 88f8f548e6a14492b229470f769fa2de, State: Running, Role: LEADER
I20260809 06:50:47.012753 25313 consensus_queue.cc:237] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } }
I20260809 06:50:47.017207 24533 catalog_manager.cc:5495] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de reported cstate change: term changed from 0 to 1, leader changed from <none> to 88f8f548e6a14492b229470f769fa2de (127.27.209.194). New cstate: current_term: 1 leader_uuid: "88f8f548e6a14492b229470f769fa2de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f8f548e6a14492b229470f769fa2de" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 34753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 } health_report { overall_health: UNKNOWN } } }
I20260809 06:50:47.041564 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:47.036437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.27.209.194:34753)
0809 06:50:47.036625 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:47.036629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:47.036706 (+ 77us) server_negotiation.cc:407] Connection header received
0809 06:50:47.036875 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:47.036878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:47.036969 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:47.037095 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:50:47.037401 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.038155 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.039522 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.039973 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.040065 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.040597 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:50:47.041088 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:50:47.041218 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:47.041328 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 06:50:47.041364 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:50:47.049774 25176 raft_consensus.cc:1261] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 1 FOLLOWER]: Refusing update from remote peer 88f8f548e6a14492b229470f769fa2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:47.049782 24753 raft_consensus.cc:1261] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 1 FOLLOWER]: Refusing update from remote peer 88f8f548e6a14492b229470f769fa2de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:47.050732 25311 consensus_queue.cc:1035] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bd358631fec461fb489fac1d07e1dda" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:47.051057 25313 consensus_queue.cc:1035] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [LEADER]: Connected to new peer: Peer: permanent_uuid: "883aa6c75aba46a9b261cae6f6bd25db" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:47.061633 25322 mvcc.cc:205] Tried to move back new op lower bound from 7316513779906117632 to 7316513779768139776. Current Snapshot: MvccSnapshot[applied={T|T < 7316513779906117632}]
I20260809 06:50:47.087664 24532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:47.087921 24532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:47.091554 24532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9240d82f752542198b5ceddb827cd473: Sending DeleteTablet for 3 replicas of tablet 8333fc768c884b0e92a9a0ac6d120763
I20260809 06:50:47.092803 24720 tablet_service.cc:1514] Processing DeleteTablet for tablet 8333fc768c884b0e92a9a0ac6d120763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:47 UTC) from {username='slave'} at 127.0.0.1:49108
I20260809 06:50:47.092917 25139 tablet_service.cc:1514] Processing DeleteTablet for tablet 8333fc768c884b0e92a9a0ac6d120763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:47 UTC) from {username='slave'} at 127.0.0.1:35466
I20260809 06:50:47.093564 24934 tablet_service.cc:1514] Processing DeleteTablet for tablet 8333fc768c884b0e92a9a0ac6d120763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:47 UTC) from {username='slave'} at 127.0.0.1:38680
I20260809 06:50:47.093797 25338 tablet_replica.cc:330] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: stopping tablet replica
I20260809 06:50:47.094265 25338 raft_consensus.cc:2229] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:47.094501 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 24638
I20260809 06:50:47.094918 25338 raft_consensus.cc:2258] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:47.097301 25338 ts_tablet_manager.cc:1905] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:47.104753 25338 ts_tablet_manager.cc:1918] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:47.104995 25338 log.cc:1196] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/8333fc768c884b0e92a9a0ac6d120763
I20260809 06:50:47.105602 25338 ts_tablet_manager.cc:1939] T 8333fc768c884b0e92a9a0ac6d120763 P 883aa6c75aba46a9b261cae6f6bd25db: Deleting consensus metadata
I20260809 06:50:47.107594 24508 catalog_manager.cc:4841] TS 883aa6c75aba46a9b261cae6f6bd25db (127.27.209.193:38519): tablet 8333fc768c884b0e92a9a0ac6d120763 (table test-table [id=2a5f026fbe67492480ad4018906b41d6]) successfully deleted
I20260809 06:50:47.108117 25340 tablet_replica.cc:330] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: stopping tablet replica
I20260809 06:50:47.108433 25341 tablet_replica.cc:330] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: stopping tablet replica
I20260809 06:50:47.108726 25340 raft_consensus.cc:2229] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:47.108821 25341 raft_consensus.cc:2229] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:50:47.109321 25340 raft_consensus.cc:2258] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:47.109467 25341 raft_consensus.cc:2258] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:47.111881 25341 ts_tablet_manager.cc:1905] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:47.112015 25340 ts_tablet_manager.cc:1905] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:47.118137 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:47.095619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.27.209.193:38519)
0809 06:50:47.096101 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:47.096107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:47.110240 (+ 14133us) server_negotiation.cc:407] Connection header received
0809 06:50:47.110526 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:47.110532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:47.110627 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:47.110700 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:47.111355 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.112462 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.114263 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.114852 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.114943 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.116487 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:47.116498 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:47.116515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:47.117277 (+ 762us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:47.117348 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:47.117373 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:47.117454 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:47.117701 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 06:50:47.117814 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260809 06:50:47.119136 25340 ts_tablet_manager.cc:1918] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:47.119223 25341 ts_tablet_manager.cc:1918] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:47.119292 24822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:47.119307 25340 log.cc:1196] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/8333fc768c884b0e92a9a0ac6d120763
I20260809 06:50:47.119376 25341 log.cc:1196] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/8333fc768c884b0e92a9a0ac6d120763
I20260809 06:50:47.119868 25340 ts_tablet_manager.cc:1939] T 8333fc768c884b0e92a9a0ac6d120763 P 8bd358631fec461fb489fac1d07e1dda: Deleting consensus metadata
I20260809 06:50:47.119952 25341 ts_tablet_manager.cc:1939] T 8333fc768c884b0e92a9a0ac6d120763 P 88f8f548e6a14492b229470f769fa2de: Deleting consensus metadata
I20260809 06:50:47.121829 24510 catalog_manager.cc:4841] TS 8bd358631fec461fb489fac1d07e1dda (127.27.209.195:40197): tablet 8333fc768c884b0e92a9a0ac6d120763 (table test-table [id=2a5f026fbe67492480ad4018906b41d6]) successfully deleted
I20260809 06:50:47.122145 24510 catalog_manager.cc:4841] TS 88f8f548e6a14492b229470f769fa2de (127.27.209.194:34753): tablet 8333fc768c884b0e92a9a0ac6d120763 (table test-table [id=2a5f026fbe67492480ad4018906b41d6]) successfully deleted
I20260809 06:50:47.206321 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 24638
I20260809 06:50:47.238133 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 24837
I20260809 06:50:47.245813 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:47.238934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.27.209.194:34753)
0809 06:50:47.239103 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:47.239107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:47.239168 (+ 61us) server_negotiation.cc:407] Connection header received
0809 06:50:47.239374 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:47.239377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:47.239485 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:47.239613 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:47.240043 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.240837 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.242709 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.243135 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.243233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.244350 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:47.244359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:47.244375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:47.245062 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:47.245126 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:47.245153 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:47.245244 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:47.245392 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 06:50:47.245469 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:50:47.246600 25027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:47.356695 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 24837
I20260809 06:50:47.381503 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 25043
I20260809 06:50:47.388253 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:47.382292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.27.209.195:40197)
0809 06:50:47.382487 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:47.382493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:47.382514 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:50:47.382672 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:47.382678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:47.382769 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:47.382871 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:47.383266 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.384091 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.385370 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:47.385767 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:47.385881 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:47.386918 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:47.386927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:47.386943 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:47.387495 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:47.387577 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:47.387604 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:47.387724 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:47.387858 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 06:50:47.387964 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:50:47.388871 25248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:47.480084 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 25043
I20260809 06:50:47.501447 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 24489
I20260809 06:50:47.502553 24592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:47.610006 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 24489
2026-08-09T06:50:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2711 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:50:47 dist-test-slave-jpj1 krb5kdc[25355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:50:47 dist-test-slave-jpj1 krb5kdc[25355](info): set up 2 sockets
Aug 09 06:50:47 dist-test-slave-jpj1 krb5kdc[25355](info): commencing operation
krb5kdc: starting...
W20260809 06:50:48.833374 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:50:49 dist-test-slave-jpj1 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:50:49.110656 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:42073
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:42073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:49.446376 25377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:49.446686 25377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:49.446831 25377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:49.454000 25377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:49.454113 25377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:49.454193 25377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:49.454273 25377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:49.454334 25377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:49.463886 25377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:42073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:42073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:49.465121 25377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:49.466387 25377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:49.477596 25386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:49.477663 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:49.477712 25393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:49.478564 25377 server_base.cc:1034] running on GCE node
I20260809 06:50:49.479317 25377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:49.480660 25377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:49.481876 25377 hybrid_clock.cc:648] HybridClock initialized: now 1786258249481822 us; error 47 us; skew 500 ppm
I20260809 06:50:49.482203 25377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:49 dist-test-slave-jpj1 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258249, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:49.485172 25377 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:50:49.486584 25377 webserver.cc:460] Webserver started at http://127.27.209.254:42225/ using document root <none> and password file <none>
I20260809 06:50:49.487223 25377 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:49.487342 25377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:49.487668 25377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:49.490553 25377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "ce0dd3ee57024551b7ff69f60119e57a"
format_stamp: "Formatted at 2026-08-09 06:50:49 on dist-test-slave-jpj1"
server_key: "2783b9f6a264c21b467215cdb53d122d"
server_key_iv: "4d20b5130631fc64ffb98fd60c05aaf9"
server_key_version: "encryptionkey@0"
I20260809 06:50:49.491274 25377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "ce0dd3ee57024551b7ff69f60119e57a"
format_stamp: "Formatted at 2026-08-09 06:50:49 on dist-test-slave-jpj1"
server_key: "2783b9f6a264c21b467215cdb53d122d"
server_key_iv: "4d20b5130631fc64ffb98fd60c05aaf9"
server_key_version: "encryptionkey@0"
I20260809 06:50:49.496145 25377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 06:50:49.499358 25401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:49.500665 25377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 06:50:49.500852 25377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "ce0dd3ee57024551b7ff69f60119e57a"
format_stamp: "Formatted at 2026-08-09 06:50:49 on dist-test-slave-jpj1"
server_key: "2783b9f6a264c21b467215cdb53d122d"
server_key_iv: "4d20b5130631fc64ffb98fd60c05aaf9"
server_key_version: "encryptionkey@0"
I20260809 06:50:49.501032 25377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:49.514551 25377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:49.518253 25377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:49.518507 25377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:49.536018 25377 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:42073
I20260809 06:50:49.536041 25482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:42073 every 8 connection(s)
I20260809 06:50:49.537410 25377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:49.541110 25483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:49.546648 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 25377
I20260809 06:50:49.546820 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:49.547276 28487 external_mini_cluster.cc:1428] Setting key 0da993dc884ee8316c583fe79f173807
I20260809 06:50:49.551813 25483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a: Bootstrap starting.
I20260809 06:50:49.555394 25483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:49.556612 25483 log.cc:824] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:49.559819 25483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a: No bootstrap required, opened a new log
Aug 09 06:50:49 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:49.569789 25483 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0dd3ee57024551b7ff69f60119e57a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42073 } }
I20260809 06:50:49.570264 25483 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:49.570371 25483 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0dd3ee57024551b7ff69f60119e57a, State: Initialized, Role: FOLLOWER
I20260809 06:50:49.571030 25483 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0dd3ee57024551b7ff69f60119e57a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42073 } }
I20260809 06:50:49.571331 25483 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:49.571482 25483 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:49.571704 25483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:49.572067 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:49.549134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.27.209.254:42073)
0809 06:50:49.549957 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:49.549968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:49.550009 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:50:49.551713 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:49.551738 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:49.552156 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:49.552474 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:49.554398 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:49.555740 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:49.557550 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:49.558387 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:49.558530 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:49.563407 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:49.563418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:49.563437 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:49.563485 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:49.567401 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:49.568128 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:49.568137 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:49.568149 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:49.568372 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:49.568783 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:49.568787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:49.568790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:49.569482 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:49.569805 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:49.570160 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 06:50:49.570622 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":371,"threads_started":1}
I20260809 06:50:49.574157 25483 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0dd3ee57024551b7ff69f60119e57a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42073 } }
I20260809 06:50:49.574617 25483 leader_election.cc:302] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0dd3ee57024551b7ff69f60119e57a; no voters:
I20260809 06:50:49.575169 25483 leader_election.cc:288] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:49.575278 25493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:49.575953 25493 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [term 1 LEADER]: Becoming Leader. State: Replica: ce0dd3ee57024551b7ff69f60119e57a, State: Running, Role: LEADER
I20260809 06:50:49.576520 25493 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0dd3ee57024551b7ff69f60119e57a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42073 } }
I20260809 06:50:49.577083 25483 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:49.579106 25495 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0dd3ee57024551b7ff69f60119e57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0dd3ee57024551b7ff69f60119e57a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42073 } } }
I20260809 06:50:49.579080 25497 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0dd3ee57024551b7ff69f60119e57a. Latest consensus state: current_term: 1 leader_uuid: "ce0dd3ee57024551b7ff69f60119e57a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0dd3ee57024551b7ff69f60119e57a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42073 } } }
I20260809 06:50:49.579504 25495 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:49.579504 25497 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:49.584584 25500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:49.589993 25500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:49.599296 25500 catalog_manager.cc:1347] Generated new cluster ID: e64d0c300a524442b1a924ddb39eceee
I20260809 06:50:49.599406 25500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:49.610525 25500 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:49.611804 25500 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:49.618742 25500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a: Generated new TSK 0
I20260809 06:50:49.619460 25500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.209.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
Entry for principal kudu/127.27.209.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab.
I20260809 06:50:49.702806 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:42073
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:50.039218 25524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:50.039496 25524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:50.039608 25524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:50.046697 25524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:50.046809 25524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:50.046931 25524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:50.056627 25524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.193.keytab
--principal=kudu/127.27.209.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:42073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:50.057916 25524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:50.059275 25524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:50.071982 25537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:50.071949 25534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:50.072650 25524 server_base.cc:1034] running on GCE node
W20260809 06:50:50.072939 25533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:50.073573 25524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:50.074501 25524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:50.075680 25524 hybrid_clock.cc:648] HybridClock initialized: now 1786258250075632 us; error 37 us; skew 500 ppm
I20260809 06:50:50.075987 25524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:50 dist-test-slave-jpj1 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:50.079109 25524 init.cc:377] Logged in from keytab as kudu/127.27.209.193@KRBTEST.COM (short username kudu)
I20260809 06:50:50.080542 25524 webserver.cc:460] Webserver started at http://127.27.209.193:33063/ using document root <none> and password file <none>
I20260809 06:50:50.081173 25524 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:50.081275 25524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:50.081535 25524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:50.084426 25524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "32c25cadb6e841b5a9c21070562a696d"
format_stamp: "Formatted at 2026-08-09 06:50:50 on dist-test-slave-jpj1"
server_key: "e5eaf53deb85bc29bf59496455232b11"
server_key_iv: "4c33c187ab2c1031373f8164c4aea234"
server_key_version: "encryptionkey@0"
I20260809 06:50:50.085160 25524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "32c25cadb6e841b5a9c21070562a696d"
format_stamp: "Formatted at 2026-08-09 06:50:50 on dist-test-slave-jpj1"
server_key: "e5eaf53deb85bc29bf59496455232b11"
server_key_iv: "4c33c187ab2c1031373f8164c4aea234"
server_key_version: "encryptionkey@0"
I20260809 06:50:50.090929 25524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 06:50:50.094610 25547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:50.095904 25524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 06:50:50.096032 25524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "32c25cadb6e841b5a9c21070562a696d"
format_stamp: "Formatted at 2026-08-09 06:50:50 on dist-test-slave-jpj1"
server_key: "e5eaf53deb85bc29bf59496455232b11"
server_key_iv: "4c33c187ab2c1031373f8164c4aea234"
server_key_version: "encryptionkey@0"
I20260809 06:50:50.096205 25524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:50.113770 25524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:50.117935 25524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:50.118196 25524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:50.119114 25524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:50.120496 25524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:50.120599 25524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:50.120719 25524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:50.120781 25524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:50.151392 25524 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:39867
I20260809 06:50:50.151438 25724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:39867 every 8 connection(s)
I20260809 06:50:50.152707 25524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:50.154603 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 25524
I20260809 06:50:50.154768 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:50:50.155278 28487 external_mini_cluster.cc:1428] Setting key cfc0df17c1af96039573634e7f09013b
Aug 09 06:50:50 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.193@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:50.174898 25734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:50.155327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:55527 (local address 127.27.209.254:42073)
0809 06:50:50.155891 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:50.155898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:50.156932 (+ 1034us) server_negotiation.cc:407] Connection header received
0809 06:50:50.159353 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:50.159358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:50.159421 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:50.159516 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:50.161482 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.162215 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.164290 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.164701 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.164813 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:50.168720 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:50.168727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:50.168731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:50.168777 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:50.171562 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:50.172468 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:50.172474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:50.172478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:50.172626 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:50.173047 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:50.173052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:50.173055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:50.173399 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:50.173538 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:50.174291 (+ 753us) server_negotiation.cc:299] Negotiation successful
0809 06:50:50.174548 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":263,"threads_started":1}
I20260809 06:50:50.175652 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:50.155983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:42073 (local address 127.27.209.193:55527)
0809 06:50:50.156785 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:50.156824 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:50.159049 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:50.159820 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:50.159832 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:50.160518 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:50.161301 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:50.161326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.162388 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.162393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:50.164139 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:50.164154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.164942 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.164949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:50.165221 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:50.165919 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:50.165955 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:50.168469 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:50.171724 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:50.171733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:50.171751 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:50.172332 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:50.172734 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:50.172737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:50.172740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:50.172930 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:50.173529 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:50.173536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:50.173884 (+ 348us) client_negotiation.cc:769] Sending connection context
0809 06:50:50.174112 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 06:50:50.174509 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":324,"threads_started":1}
I20260809 06:50:50.177191 25728 heartbeater.cc:347] Connected to a master server at 127.27.209.254:42073
I20260809 06:50:50.177488 25728 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:50.178226 25728 heartbeater.cc:511] Master 127.27.209.254:42073 requested a full tablet report, sending...
I20260809 06:50:50.180315 25424 ts_manager.cc:194] Registered new tserver with Master: 32c25cadb6e841b5a9c21070562a696d (127.27.209.193:39867)
I20260809 06:50:50.181772 25424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.193@KRBTEST.COM'} at 127.27.209.193:55527
WARNING: no policy specified for kudu/127.27.209.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
Entry for principal kudu/127.27.209.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab.
I20260809 06:50:50.222132 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:42073
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:50.557407 25742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:50.557693 25742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:50.557834 25742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:50.564968 25742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:50.565090 25742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:50.565243 25742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:50.574592 25742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.194.keytab
--principal=kudu/127.27.209.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:42073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:50.575831 25742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:50.577083 25742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:50.589340 25748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:50.589318 25749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:50.589378 25751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:50.590359 25742 server_base.cc:1034] running on GCE node
I20260809 06:50:50.590895 25742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:50.591845 25742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:50.593052 25742 hybrid_clock.cc:648] HybridClock initialized: now 1786258250592997 us; error 45 us; skew 500 ppm
I20260809 06:50:50.593364 25742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:50 dist-test-slave-jpj1 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:50.599462 25742 init.cc:377] Logged in from keytab as kudu/127.27.209.194@KRBTEST.COM (short username kudu)
I20260809 06:50:50.600857 25742 webserver.cc:460] Webserver started at http://127.27.209.194:45977/ using document root <none> and password file <none>
I20260809 06:50:50.601485 25742 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:50.601584 25742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:50.601845 25742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:50.604709 25742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "41b31b98f6c641bf9a90de6ab08e21e4"
format_stamp: "Formatted at 2026-08-09 06:50:50 on dist-test-slave-jpj1"
server_key: "360e829e864e9f9781ebdf8c738ea1bd"
server_key_iv: "53816903a12b693c268747a188a9528c"
server_key_version: "encryptionkey@0"
I20260809 06:50:50.605443 25742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "41b31b98f6c641bf9a90de6ab08e21e4"
format_stamp: "Formatted at 2026-08-09 06:50:50 on dist-test-slave-jpj1"
server_key: "360e829e864e9f9781ebdf8c738ea1bd"
server_key_iv: "53816903a12b693c268747a188a9528c"
server_key_version: "encryptionkey@0"
I20260809 06:50:50.610311 25742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 06:50:50.613516 25770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:50.614737 25742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:50:50.614871 25742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "41b31b98f6c641bf9a90de6ab08e21e4"
format_stamp: "Formatted at 2026-08-09 06:50:50 on dist-test-slave-jpj1"
server_key: "360e829e864e9f9781ebdf8c738ea1bd"
server_key_iv: "53816903a12b693c268747a188a9528c"
server_key_version: "encryptionkey@0"
I20260809 06:50:50.615024 25742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:50.654930 25742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:50.658619 25742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:50.658855 25742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:50.659806 25742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:50.661072 25742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:50.661144 25742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:50.661260 25742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:50.661306 25742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:50.691275 25742 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:44601
I20260809 06:50:50.691323 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:44601 every 8 connection(s)
I20260809 06:50:50.692553 25742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:50.700382 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 25742
I20260809 06:50:50.700559 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:50:50.701069 28487 external_mini_cluster.cc:1428] Setting key 1c24a8b4ac64b5bdabc1f5a659a48b97
Aug 09 06:50:50 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258250, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.194@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:50.713042 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:50.694989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:45281 (local address 127.27.209.254:42073)
0809 06:50:50.695455 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:50.695461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:50.696459 (+ 998us) server_negotiation.cc:407] Connection header received
0809 06:50:50.698222 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:50.698227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:50.698294 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:50.698393 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:50.699939 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.701069 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.703040 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.703664 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.703822 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:50.707295 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:50.707301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:50.707306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:50.707341 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:50.710193 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:50.710846 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:50.710851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:50.710855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:50.711007 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:50.711378 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:50.711382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:50.711385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:50.711752 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:50.711887 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:50.712479 (+ 592us) server_negotiation.cc:299] Negotiation successful
0809 06:50:50.712679 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20260809 06:50:50.713848 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:50.695632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:42073 (local address 127.27.209.194:45281)
0809 06:50:50.696301 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:50.696347 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:50.697957 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:50.698574 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:50.698582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:50.699089 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:50.699761 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:50.699782 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.701252 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.701258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:50.702893 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:50.702908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:50.703911 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:50.703919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:50.704198 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:50.704905 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:50.704928 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:50.707080 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:50.710316 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:50.710323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:50.710334 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:50.710727 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:50.711103 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:50.711106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:50.711108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:50.711283 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:50.711862 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:50.711867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:50.712150 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 06:50:50.712371 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 06:50:50.712731 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":257,"threads_started":1}
I20260809 06:50:50.715214 25951 heartbeater.cc:347] Connected to a master server at 127.27.209.254:42073
I20260809 06:50:50.715505 25951 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:50.716228 25951 heartbeater.cc:511] Master 127.27.209.254:42073 requested a full tablet report, sending...
I20260809 06:50:50.717694 25424 ts_manager.cc:194] Registered new tserver with Master: 41b31b98f6c641bf9a90de6ab08e21e4 (127.27.209.194:44601)
I20260809 06:50:50.718504 25424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.194@KRBTEST.COM'} at 127.27.209.194:45281
WARNING: no policy specified for kudu/127.27.209.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
Entry for principal kudu/127.27.209.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab.
I20260809 06:50:50.764451 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:42073
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:51.090921 25961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:51.091229 25961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:51.091367 25961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:51.099668 25961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:51.099800 25961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:51.099965 25961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:51.109901 25961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.195.keytab
--principal=kudu/127.27.209.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:42073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:51.111176 25961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:51.112558 25961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:51.125484 25970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:51.125522 25977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:51.125547 25971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:51.126370 25961 server_base.cc:1034] running on GCE node
I20260809 06:50:51.126987 25961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:51.127904 25961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:51.129122 25961 hybrid_clock.cc:648] HybridClock initialized: now 1786258251129060 us; error 43 us; skew 500 ppm
I20260809 06:50:51.129423 25961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:51 dist-test-slave-jpj1 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258251, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:51.132488 25961 init.cc:377] Logged in from keytab as kudu/127.27.209.195@KRBTEST.COM (short username kudu)
I20260809 06:50:51.133859 25961 webserver.cc:460] Webserver started at http://127.27.209.195:36661/ using document root <none> and password file <none>
I20260809 06:50:51.134487 25961 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:51.134588 25961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:51.134840 25961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:51.137611 25961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "a87e4d75635548b8a8349826af343599"
format_stamp: "Formatted at 2026-08-09 06:50:51 on dist-test-slave-jpj1"
server_key: "3b8030a30d9ae542ee2e9e14f4332f21"
server_key_iv: "4769cd1c0bc44dc9a494771b4f757378"
server_key_version: "encryptionkey@0"
I20260809 06:50:51.138319 25961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "a87e4d75635548b8a8349826af343599"
format_stamp: "Formatted at 2026-08-09 06:50:51 on dist-test-slave-jpj1"
server_key: "3b8030a30d9ae542ee2e9e14f4332f21"
server_key_iv: "4769cd1c0bc44dc9a494771b4f757378"
server_key_version: "encryptionkey@0"
I20260809 06:50:51.143261 25961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:50:51.146546 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:51.147821 25961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:50:51.147961 25961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "a87e4d75635548b8a8349826af343599"
format_stamp: "Formatted at 2026-08-09 06:50:51 on dist-test-slave-jpj1"
server_key: "3b8030a30d9ae542ee2e9e14f4332f21"
server_key_iv: "4769cd1c0bc44dc9a494771b4f757378"
server_key_version: "encryptionkey@0"
I20260809 06:50:51.148128 25961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:51.177476 25961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:51.181623 25961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:51.181898 25961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:51.182833 25961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:51.184171 25961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:51.184253 25961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:51.184366 25961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:51.184423 25961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:51.184818 25728 heartbeater.cc:503] Master 127.27.209.254:42073 was elected leader, sending a full tablet report...
I20260809 06:50:51.213925 25961 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:37345
I20260809 06:50:51.213967 26161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:37345 every 8 connection(s)
I20260809 06:50:51.215171 25961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:51.220613 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 25961
I20260809 06:50:51.220789 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:50:51.221210 28487 external_mini_cluster.cc:1428] Setting key 11aa1a8927b0cf68c404b43ede19050b
Aug 09 06:50:51 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258251, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.195@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:50:51.234886 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.217716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:54423 (local address 127.27.209.254:42073)
0809 06:50:51.218177 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.218181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.219172 (+ 991us) server_negotiation.cc:407] Connection header received
0809 06:50:51.221133 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.221138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.221233 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.221331 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:51.222910 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.224074 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.225626 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.226030 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.226141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.229420 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:51.229426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:51.229429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:51.229468 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:51.231904 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.232646 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.232651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.232653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.232787 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.233200 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.233205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.233208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.233468 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:51.233607 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.234269 (+ 662us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.234492 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260809 06:50:51.235929 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.218322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:42073 (local address 127.27.209.195:54423)
0809 06:50:51.219017 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.219061 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:51.220861 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:51.221580 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:51.221588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:51.222121 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:51.222749 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.222767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.224226 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.224230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.225483 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.225500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.226258 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.226266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.226490 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.227097 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:50:51.227118 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:51.229219 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:51.232033 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:51.232042 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:51.232058 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:51.232531 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:51.232890 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:50:51.232895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:50:51.232898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:50:51.233083 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:50:51.233592 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:51.233600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:51.233916 (+ 316us) client_negotiation.cc:769] Sending connection context
0809 06:50:51.234154 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 06:50:51.234651 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":255,"threads_started":1}
I20260809 06:50:51.237298 26162 heartbeater.cc:347] Connected to a master server at 127.27.209.254:42073
I20260809 06:50:51.237579 26162 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:51.238262 26162 heartbeater.cc:511] Master 127.27.209.254:42073 requested a full tablet report, sending...
I20260809 06:50:51.239643 25424 ts_manager.cc:194] Registered new tserver with Master: a87e4d75635548b8a8349826af343599 (127.27.209.195:37345)
I20260809 06:50:51.240424 25424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.209.195@KRBTEST.COM'} at 127.27.209.195:54423
I20260809 06:50:51.248386 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:50:51.264569 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.252441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.27.209.254:42073)
0809 06:50:51.252605 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.252612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.253011 (+ 399us) server_negotiation.cc:407] Connection header received
0809 06:50:51.253169 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.253173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.253235 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.253318 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:51.254733 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.255451 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.256772 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.257183 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.257297 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.259523 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:51.259528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:51.259558 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:51.259597 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:51.261974 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.262638 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.262644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.262648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.262781 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.263138 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.263143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.263146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.263459 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:51.263713 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.264048 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.264266 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:50:51.268134 25424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:50:51.274832 25424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:50:51.304477 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.287047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.27.209.195:37345)
0809 06:50:51.296220 (+ 9173us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.296230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.296267 (+ 37us) server_negotiation.cc:407] Connection header received
0809 06:50:51.296366 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.296375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.296638 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.296864 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:51.298234 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.299988 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.302440 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.303322 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.303497 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.303711 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.303986 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.304146 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8984,"thread_start_us":8846,"threads_started":1}
I20260809 06:50:51.304540 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.286802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37345 (local address 127.0.0.1:53936)
0809 06:50:51.293961 (+ 7159us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.293998 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:51.294170 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:51.297445 (+ 3275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:51.297454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:51.297489 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:51.297984 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.298006 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.300144 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.300149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.302296 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.302311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.303503 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.303509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.303722 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.303773 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 06:50:51.303898 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 06:50:51.304082 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6908,"mutex_wait_us":1,"thread_start_us":6774,"threads_started":1}
I20260809 06:50:51.305472 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.286943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:44601 (local address 127.0.0.1:36196)
0809 06:50:51.295455 (+ 8512us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.295471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:51.295618 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:51.297692 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:51.297697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:51.297721 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:51.298225 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.298240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.299831 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.299836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.302445 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.302461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.304617 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.304624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.304801 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.304831 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 06:50:51.304927 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 06:50:51.305044 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8396,"mutex_wait_us":60,"thread_start_us":8274,"threads_started":1}
I20260809 06:50:51.306327 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.287395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.27.209.194:44601)
0809 06:50:51.297095 (+ 9700us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.297101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.297123 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:50:51.297204 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.297212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.297459 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.298416 (+ 957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:51.298593 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.299663 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.302634 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.304130 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.304293 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.304461 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.305946 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.306086 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9553,"thread_start_us":9423,"threads_started":1}
I20260809 06:50:51.309440 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.296592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42684)
0809 06:50:51.300289 (+ 3697us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.300311 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:51.300517 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:51.301373 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:51.301378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:51.301402 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:51.301836 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.301848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.303784 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.303789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.307383 (+ 3594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.307396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.308707 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.308722 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.308884 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.308916 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:50:51.308995 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 06:50:51.309117 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3543,"thread_start_us":3388,"threads_started":1}
I20260809 06:50:51.309527 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.295937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.27.209.193:39867)
0809 06:50:51.296660 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.296669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.300405 (+ 3736us) server_negotiation.cc:407] Connection header received
0809 06:50:51.300697 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.300705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.300919 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.301097 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:51.302023 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.303579 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.307556 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.308494 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.308707 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.308871 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.309047 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.309223 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":338,"threads_started":1}
I20260809 06:50:51.308949 26047 tablet_service.cc:1467] Processing CreateTablet for tablet 3eceb8e17f404c2b99740f082fd2b015 (DEFAULT_TABLE table=test-table [id=8c4d2467b93740f28aebecf7ee3fd9cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:51.309873 25846 tablet_service.cc:1467] Processing CreateTablet for tablet 3eceb8e17f404c2b99740f082fd2b015 (DEFAULT_TABLE table=test-table [id=8c4d2467b93740f28aebecf7ee3fd9cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:51.310627 26047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eceb8e17f404c2b99740f082fd2b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:51.311376 25846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eceb8e17f404c2b99740f082fd2b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:51.312821 25619 tablet_service.cc:1467] Processing CreateTablet for tablet 3eceb8e17f404c2b99740f082fd2b015 (DEFAULT_TABLE table=test-table [id=8c4d2467b93740f28aebecf7ee3fd9cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:51.313894 25619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eceb8e17f404c2b99740f082fd2b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:51.324939 26189 tablet_bootstrap.cc:492] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Bootstrap starting.
I20260809 06:50:51.326992 26192 tablet_bootstrap.cc:492] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Bootstrap starting.
I20260809 06:50:51.327813 26190 tablet_bootstrap.cc:492] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Bootstrap starting.
I20260809 06:50:51.328771 26189 tablet_bootstrap.cc:654] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:51.330101 26189 log.cc:824] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:51.331135 26192 tablet_bootstrap.cc:654] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:51.331210 26190 tablet_bootstrap.cc:654] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:51.332391 26190 log.cc:824] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:51.332391 26192 log.cc:824] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:51.333542 26189 tablet_bootstrap.cc:492] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: No bootstrap required, opened a new log
I20260809 06:50:51.333834 26189 ts_tablet_manager.cc:1397] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260809 06:50:51.338515 26190 tablet_bootstrap.cc:492] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: No bootstrap required, opened a new log
I20260809 06:50:51.338797 26190 ts_tablet_manager.cc:1397] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20260809 06:50:51.340260 26192 tablet_bootstrap.cc:492] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: No bootstrap required, opened a new log
I20260809 06:50:51.340494 26192 ts_tablet_manager.cc:1397] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260809 06:50:51.344122 26189 raft_consensus.cc:348] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.344566 26189 raft_consensus.cc:374] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:51.344674 26189 raft_consensus.cc:729] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c25cadb6e841b5a9c21070562a696d, State: Initialized, Role: FOLLOWER
I20260809 06:50:51.345299 26189 consensus_queue.cc:260] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.347005 26189 ts_tablet_manager.cc:1428] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Time spent starting tablet: real 0.013s user 0.008s sys 0.007s
I20260809 06:50:51.348760 26190 raft_consensus.cc:348] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.349184 26190 raft_consensus.cc:374] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:51.349033 26192 raft_consensus.cc:348] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.349287 26190 raft_consensus.cc:729] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87e4d75635548b8a8349826af343599, State: Initialized, Role: FOLLOWER
I20260809 06:50:51.349406 26192 raft_consensus.cc:374] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:51.349504 26192 raft_consensus.cc:729] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b31b98f6c641bf9a90de6ab08e21e4, State: Initialized, Role: FOLLOWER
I20260809 06:50:51.350024 26190 consensus_queue.cc:260] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.350114 26192 consensus_queue.cc:260] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.351104 26162 heartbeater.cc:503] Master 127.27.209.254:42073 was elected leader, sending a full tablet report...
I20260809 06:50:51.351895 26190 ts_tablet_manager.cc:1428] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260809 06:50:51.352346 25951 heartbeater.cc:503] Master 127.27.209.254:42073 was elected leader, sending a full tablet report...
I20260809 06:50:51.352978 26192 ts_tablet_manager.cc:1428] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
W20260809 06:50:51.405074 25733 tablet.cc:2369] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:51.427738 26196 raft_consensus.cc:482] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:51.428037 26196 raft_consensus.cc:504] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.429803 26196 leader_election.cc:288] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a87e4d75635548b8a8349826af343599 (127.27.209.195:37345), 41b31b98f6c641bf9a90de6ab08e21e4 (127.27.209.194:44601)
I20260809 06:50:51.438395 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.430293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:42431 (local address 127.27.209.194:44601)
0809 06:50:51.430497 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.430505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.431227 (+ 722us) server_negotiation.cc:407] Connection header received
0809 06:50:51.431424 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.431430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.431507 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.431624 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:51.432339 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.433456 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.436164 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.437391 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.437526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.437600 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.438008 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.438137 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:50:51.438393 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.430364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:38037 (local address 127.27.209.195:37345)
0809 06:50:51.430536 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.430541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.431228 (+ 687us) server_negotiation.cc:407] Connection header received
0809 06:50:51.431424 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.431430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.431508 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.431624 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:51.432392 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.433471 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.436164 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.437370 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.437531 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.437600 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.438008 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.438137 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:50:51.438625 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.430150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37345 (local address 127.27.209.193:38037)
0809 06:50:51.431115 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.431128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:51.431289 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:51.431636 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:51.431641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:51.431673 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:51.432241 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.432251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.433591 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.433595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.436000 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.436017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.437592 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.437599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.437818 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.437889 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 06:50:51.437977 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 06:50:51.438147 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":173,"spinlock_wait_cycles":45312,"thread_start_us":519,"threads_started":1}
I20260809 06:50:51.438647 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.430062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:44601 (local address 127.27.209.193:42431)
0809 06:50:51.431094 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.431118 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:51.431282 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:51.431636 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:51.431641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:51.431682 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:51.432188 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.432206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.433589 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.433595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.436000 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:51.436017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.437622 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.437628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:51.437820 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.437882 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 06:50:51.437977 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 06:50:51.438147 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":133,"spinlock_wait_cycles":4992,"thread_start_us":735,"threads_started":1}
I20260809 06:50:51.439646 25879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eceb8e17f404c2b99740f082fd2b015" candidate_uuid: "32c25cadb6e841b5a9c21070562a696d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" is_pre_election: true
I20260809 06:50:51.439771 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eceb8e17f404c2b99740f082fd2b015" candidate_uuid: "32c25cadb6e841b5a9c21070562a696d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87e4d75635548b8a8349826af343599" is_pre_election: true
I20260809 06:50:51.440124 25879 raft_consensus.cc:2454] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c25cadb6e841b5a9c21070562a696d in term 0.
I20260809 06:50:51.440229 26078 raft_consensus.cc:2454] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32c25cadb6e841b5a9c21070562a696d in term 0.
I20260809 06:50:51.440781 25550 leader_election.cc:302] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32c25cadb6e841b5a9c21070562a696d, 41b31b98f6c641bf9a90de6ab08e21e4, a87e4d75635548b8a8349826af343599; no voters:
I20260809 06:50:51.441205 26196 raft_consensus.cc:2781] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:50:51.441349 26196 raft_consensus.cc:482] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:50:51.441443 26196 raft_consensus.cc:3037] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:51.443588 26196 raft_consensus.cc:504] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.444142 26196 leader_election.cc:288] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [CANDIDATE]: Term 1 election: Requested vote from peers a87e4d75635548b8a8349826af343599 (127.27.209.195:37345), 41b31b98f6c641bf9a90de6ab08e21e4 (127.27.209.194:44601)
W20260809 06:50:51.444545 25952 tablet.cc:2369] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:51.444553 26078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eceb8e17f404c2b99740f082fd2b015" candidate_uuid: "32c25cadb6e841b5a9c21070562a696d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87e4d75635548b8a8349826af343599"
I20260809 06:50:51.444646 25879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eceb8e17f404c2b99740f082fd2b015" candidate_uuid: "32c25cadb6e841b5a9c21070562a696d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41b31b98f6c641bf9a90de6ab08e21e4"
I20260809 06:50:51.444743 26078 raft_consensus.cc:3037] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:51.444846 25879 raft_consensus.cc:3037] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:51.447909 25879 raft_consensus.cc:2454] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c25cadb6e841b5a9c21070562a696d in term 1.
I20260809 06:50:51.447909 26078 raft_consensus.cc:2454] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32c25cadb6e841b5a9c21070562a696d in term 1.
I20260809 06:50:51.448280 25550 leader_election.cc:302] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c25cadb6e841b5a9c21070562a696d, 41b31b98f6c641bf9a90de6ab08e21e4; no voters:
I20260809 06:50:51.448596 26196 raft_consensus.cc:2781] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:51.449020 26196 raft_consensus.cc:686] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 1 LEADER]: Becoming Leader. State: Replica: 32c25cadb6e841b5a9c21070562a696d, State: Running, Role: LEADER
I20260809 06:50:51.449525 26196 consensus_queue.cc:237] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } }
I20260809 06:50:51.453722 25424 catalog_manager.cc:5495] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32c25cadb6e841b5a9c21070562a696d (127.27.209.193). New cstate: current_term: 1 leader_uuid: "32c25cadb6e841b5a9c21070562a696d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32c25cadb6e841b5a9c21070562a696d" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 39867 } health_report { overall_health: HEALTHY } } }
W20260809 06:50:51.466814 26165 tablet.cc:2369] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:51.495954 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.491054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.27.209.193:39867)
0809 06:50:51.491244 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.491248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.491265 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:50:51.491350 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.491354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.491431 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.491592 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:50:51.491912 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.492616 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.493928 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.494309 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.494393 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.494921 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:50:51.495414 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:50:51.495526 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.495683 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.495749 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:50:51.504033 26078 raft_consensus.cc:1261] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 1 FOLLOWER]: Refusing update from remote peer 32c25cadb6e841b5a9c21070562a696d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:51.504022 25879 raft_consensus.cc:1261] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 1 FOLLOWER]: Refusing update from remote peer 32c25cadb6e841b5a9c21070562a696d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:51.504979 26207 consensus_queue.cc:1035] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b31b98f6c641bf9a90de6ab08e21e4" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:51.505307 26196 consensus_queue.cc:1035] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a87e4d75635548b8a8349826af343599" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:51.528080 25422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:51.528353 25422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:51.531876 25422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce0dd3ee57024551b7ff69f60119e57a: Sending DeleteTablet for 3 replicas of tablet 3eceb8e17f404c2b99740f082fd2b015
I20260809 06:50:51.532914 26047 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eceb8e17f404c2b99740f082fd2b015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:51 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:53936
I20260809 06:50:51.533154 25846 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eceb8e17f404c2b99740f082fd2b015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:51 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:36196
I20260809 06:50:51.534029 25619 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eceb8e17f404c2b99740f082fd2b015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:51 UTC) from {username='kudu', principal='kudu/127.27.209.254@KRBTEST.COM'} at 127.0.0.1:42684
I20260809 06:50:51.534718 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 25524
I20260809 06:50:51.544090 26225 tablet_replica.cc:330] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: stopping tablet replica
I20260809 06:50:51.545094 26225 raft_consensus.cc:2229] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:51.546250 26225 raft_consensus.cc:2258] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:51.549180 26227 tablet_replica.cc:330] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: stopping tablet replica
I20260809 06:50:51.549639 26227 raft_consensus.cc:2229] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:50:51.550340 26227 raft_consensus.cc:2258] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:51.556978 26227 ts_tablet_manager.cc:1905] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:51.559746 26225 ts_tablet_manager.cc:1905] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:51.560336 26224 tablet_replica.cc:330] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: stopping tablet replica
I20260809 06:50:51.560760 26224 raft_consensus.cc:2229] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:51.561277 26224 raft_consensus.cc:2258] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:51.563272 26224 ts_tablet_manager.cc:1905] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 06:50:51 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.193@KRBTEST.COM
I20260809 06:50:51.568521 26225 ts_tablet_manager.cc:1918] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:51.569918 26224 ts_tablet_manager.cc:1918] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:51.570143 26224 log.cc:1196] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/3eceb8e17f404c2b99740f082fd2b015
I20260809 06:50:51.570760 26224 ts_tablet_manager.cc:1939] T 3eceb8e17f404c2b99740f082fd2b015 P a87e4d75635548b8a8349826af343599: Deleting consensus metadata
I20260809 06:50:51.572752 25403 catalog_manager.cc:4841] TS a87e4d75635548b8a8349826af343599 (127.27.209.195:37345): tablet 3eceb8e17f404c2b99740f082fd2b015 (table test-table [id=8c4d2467b93740f28aebecf7ee3fd9cc]) successfully deleted
I20260809 06:50:51.575390 26225 log.cc:1196] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/3eceb8e17f404c2b99740f082fd2b015
I20260809 06:50:51.575901 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.535753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.27.209.193:39867)
0809 06:50:51.535933 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.535940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.545970 (+ 10030us) server_negotiation.cc:407] Connection header received
0809 06:50:51.546440 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.546447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.546534 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.546618 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:51.550800 (+ 4182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.552115 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.554430 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.555134 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.555252 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.567721 (+ 12469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:51.567736 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:51.567755 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:51.567811 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:51.572245 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.573156 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.573165 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.573180 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.573352 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.573802 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.573807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.573810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.574301 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:51.574631 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.575232 (+ 601us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.575483 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:50:51.576300 26225 ts_tablet_manager.cc:1939] T 3eceb8e17f404c2b99740f082fd2b015 P 41b31b98f6c641bf9a90de6ab08e21e4: Deleting consensus metadata
I20260809 06:50:51.576506 26227 ts_tablet_manager.cc:1918] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:51.577015 26227 log.cc:1196] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786257935933888-28487-0/minicluster-data/ts-0/wal/wals/3eceb8e17f404c2b99740f082fd2b015
I20260809 06:50:51.577384 25719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:51.579160 25404 catalog_manager.cc:4841] TS 41b31b98f6c641bf9a90de6ab08e21e4 (127.27.209.194:44601): tablet 3eceb8e17f404c2b99740f082fd2b015 (table test-table [id=8c4d2467b93740f28aebecf7ee3fd9cc]) successfully deleted
I20260809 06:50:51.674415 26227 ts_tablet_manager.cc:1939] T 3eceb8e17f404c2b99740f082fd2b015 P 32c25cadb6e841b5a9c21070562a696d: Deleting consensus metadata
I20260809 06:50:51.675180 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 25524
I20260809 06:50:51.697965 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 25742
W20260809 06:50:51.697901 25404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.209.193:39867 (error 108)
I20260809 06:50:51.699961 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.699409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42704)
0809 06:50:51.699663 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.699797 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260809 06:50:51.700210 25404 catalog_manager.cc:4584] TS 32c25cadb6e841b5a9c21070562a696d (127.27.209.193:39867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3eceb8e17f404c2b99740f082fd2b015: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Aug 09 06:50:51 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.194@KRBTEST.COM
I20260809 06:50:51.713624 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.698984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.27.209.194:44601)
0809 06:50:51.699148 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.699154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.699173 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:50:51.699310 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.699315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.699380 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.699473 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:51.700967 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.701750 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.702967 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.703395 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.703505 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.707883 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:51.707894 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:51.707907 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:51.707944 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:51.710864 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.711624 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.711633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.711637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.711763 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.712131 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.712136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.712140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.712510 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:51.712793 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.713087 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.713308 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:50:51.714280 25943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:51.753979 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.753403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42706)
0809 06:50:51.753637 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.753772 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 06:50:51.755044 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.754651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42712)
0809 06:50:51.754805 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.754890 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 06:50:51.802584 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 25742
I20260809 06:50:51.816282 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.815770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42724)
0809 06:50:51.815982 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.816092 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 06:50:51.817346 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.816948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42736)
0809 06:50:51.817113 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.817211 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 06:50:51.824836 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 25961
Aug 09 06:50:51 dist-test-slave-jpj1 krb5kdc[25355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.195@KRBTEST.COM
I20260809 06:50:51.839700 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.825542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.27.209.195:37345)
0809 06:50:51.825718 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:51.825724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:51.825743 (+ 19us) server_negotiation.cc:407] Connection header received
0809 06:50:51.825916 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:51.825920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:51.825985 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:51.826081 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:51.827420 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.828183 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.829491 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:51.829907 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:51.830017 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:51.834505 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:51.834522 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:51.834539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:51.834574 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:51.837017 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.837665 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.837673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.837675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.837811 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:51.838164 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:51.838168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:51.838170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:51.838542 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:51.838822 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:51.839148 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 06:50:51.839378 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:50:51.840296 26156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:51.897683 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.897181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42752)
0809 06:50:51.897402 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.897498 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 06:50:51.898747 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:51.898345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:39867 (local address 127.0.0.1:42758)
0809 06:50:51.898550 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:51.898628 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 06:50:51.926339 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 25961
I20260809 06:50:51.949653 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 25377
I20260809 06:50:51.950703 25477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:52.060154 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 25377
2026-08-09T06:50:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4451 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T06:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:52Z Disabled control of system clock
I20260809 06:50:52.151259 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44439
--webserver_interface=127.27.209.254
--webserver_port=0
--builtin_ntp_servers=127.27.209.212:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:50:52.486521 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:52.486824 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:52.486912 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:52.494060 26245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:52.494174 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:52.494211 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:52.494279 26245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:52.494323 26245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:52.503310 26245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:52.504590 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:52.506443 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:52.518055 26254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:52.518016 26255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:52.518100 26257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:52.519236 26245 server_base.cc:1034] running on GCE node
I20260809 06:50:52.520058 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:52.521463 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:52.522773 26245 hybrid_clock.cc:648] HybridClock initialized: now 1786258252522694 us; error 50 us; skew 500 ppm
I20260809 06:50:52.523142 26245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:52.525393 26245 webserver.cc:460] Webserver started at http://127.27.209.254:46453/ using document root <none> and password file <none>
I20260809 06:50:52.526070 26245 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:52.526187 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:52.526494 26245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:52.529405 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "292d77ebe0584725bbe9854e30c0b3c2"
format_stamp: "Formatted at 2026-08-09 06:50:52 on dist-test-slave-jpj1"
server_key: "f80697c0f1b00382154ec5a1fb09a44b"
server_key_iv: "c53510c31bf7532e42f774ca1f4842b6"
server_key_version: "encryptionkey@0"
I20260809 06:50:52.530229 26245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "292d77ebe0584725bbe9854e30c0b3c2"
format_stamp: "Formatted at 2026-08-09 06:50:52 on dist-test-slave-jpj1"
server_key: "f80697c0f1b00382154ec5a1fb09a44b"
server_key_iv: "c53510c31bf7532e42f774ca1f4842b6"
server_key_version: "encryptionkey@0"
I20260809 06:50:52.535266 26245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 06:50:52.538702 26266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:52.540189 26245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:52.540365 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "292d77ebe0584725bbe9854e30c0b3c2"
format_stamp: "Formatted at 2026-08-09 06:50:52 on dist-test-slave-jpj1"
server_key: "f80697c0f1b00382154ec5a1fb09a44b"
server_key_iv: "c53510c31bf7532e42f774ca1f4842b6"
server_key_version: "encryptionkey@0"
I20260809 06:50:52.540562 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:52.555306 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:52.556185 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:52.556458 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:52.573855 26245 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:44439
I20260809 06:50:52.573880 26348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:44439 every 8 connection(s)
I20260809 06:50:52.575174 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:52.577101 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 26245
I20260809 06:50:52.577261 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:52.577701 28487 external_mini_cluster.cc:1428] Setting key d22cbdeadb9a29a83f64ef8bd1238e61
I20260809 06:50:52.579082 26349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:52.592702 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:52.579508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.27.209.254:44439)
0809 06:50:52.580596 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:52.580611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:52.580664 (+ 53us) server_negotiation.cc:407] Connection header received
0809 06:50:52.582506 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:52.582533 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:52.583143 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:52.583589 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:52.584150 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:52.586151 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:52.587602 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:52.588591 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:52.588787 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:52.589907 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:52.589918 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:52.589935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:52.590638 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:52.590752 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:52.590783 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:52.590904 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:52.591129 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 06:50:52.591455 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":342,"threads_started":1}
I20260809 06:50:52.593643 26349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2: Bootstrap starting.
I20260809 06:50:52.596648 26349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:52.597796 26349 log.cc:824] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:52.600556 26349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2: No bootstrap required, opened a new log
I20260809 06:50:52.607568 26349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292d77ebe0584725bbe9854e30c0b3c2" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44439 } }
I20260809 06:50:52.607918 26349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:52.608006 26349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292d77ebe0584725bbe9854e30c0b3c2, State: Initialized, Role: FOLLOWER
I20260809 06:50:52.608505 26349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292d77ebe0584725bbe9854e30c0b3c2" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44439 } }
I20260809 06:50:52.608716 26349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:52.608837 26349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:52.608990 26349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:52.611034 26349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292d77ebe0584725bbe9854e30c0b3c2" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44439 } }
I20260809 06:50:52.611618 26349 leader_election.cc:302] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 292d77ebe0584725bbe9854e30c0b3c2; no voters:
I20260809 06:50:52.612211 26349 leader_election.cc:288] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:52.612334 26354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:52.613071 26354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [term 1 LEADER]: Becoming Leader. State: Replica: 292d77ebe0584725bbe9854e30c0b3c2, State: Running, Role: LEADER
I20260809 06:50:52.613683 26354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292d77ebe0584725bbe9854e30c0b3c2" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44439 } }
I20260809 06:50:52.614156 26349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:52.616401 26356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 292d77ebe0584725bbe9854e30c0b3c2. Latest consensus state: current_term: 1 leader_uuid: "292d77ebe0584725bbe9854e30c0b3c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292d77ebe0584725bbe9854e30c0b3c2" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44439 } } }
I20260809 06:50:52.616824 26356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:52.617367 26355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "292d77ebe0584725bbe9854e30c0b3c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "292d77ebe0584725bbe9854e30c0b3c2" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44439 } } }
I20260809 06:50:52.617692 26355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:52.618513 26366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:52.623250 26366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:52.630826 26366 catalog_manager.cc:1347] Generated new cluster ID: ecefd71c558f47a08ac6d1d46bb1a2ca
I20260809 06:50:52.630937 26366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:52.640242 26366 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:52.641388 26366 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:52.656522 26366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2: Generated new TSK 0
I20260809 06:50:52.657295 26366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:50:52.667083 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.193:0
--local_ip_for_outbound_sockets=127.27.209.193
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44439
--builtin_ntp_servers=127.27.209.212:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 06:50:52.997088 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:52.997342 26377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:52.997467 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:53.004462 26377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:53.004565 26377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:53.004721 26377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.193
I20260809 06:50:53.013836 26377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.209.193
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.193
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:53.015205 26377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:53.017102 26377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:53.029744 26383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:53.029812 26384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:53.029840 26390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:53.030679 26377 server_base.cc:1034] running on GCE node
I20260809 06:50:53.031291 26377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:53.032220 26377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:53.033419 26377 hybrid_clock.cc:648] HybridClock initialized: now 1786258253033361 us; error 49 us; skew 500 ppm
I20260809 06:50:53.033727 26377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:53.035902 26377 webserver.cc:460] Webserver started at http://127.27.209.193:40513/ using document root <none> and password file <none>
I20260809 06:50:53.036648 26377 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:53.036755 26377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:53.037056 26377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:53.039909 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data/instance:
uuid: "2dd56bebbdf245e4904bc62e121ea648"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "f41ea347c55438f2d56f4b9c10db0d12"
server_key_iv: "3a567cfbe81d288c727add014c969ab9"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.040684 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance:
uuid: "2dd56bebbdf245e4904bc62e121ea648"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "f41ea347c55438f2d56f4b9c10db0d12"
server_key_iv: "3a567cfbe81d288c727add014c969ab9"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.045713 26377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:50:53.049168 26397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.050427 26377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:50:53.050575 26377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal
uuid: "2dd56bebbdf245e4904bc62e121ea648"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "f41ea347c55438f2d56f4b9c10db0d12"
server_key_iv: "3a567cfbe81d288c727add014c969ab9"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.050781 26377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:53.076031 26377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:53.076892 26377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:53.077200 26377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:53.078190 26377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:53.079692 26377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:53.079789 26377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.079905 26377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:53.079977 26377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.110942 26377 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.193:38501
I20260809 06:50:53.111011 26578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.193:38501 every 8 connection(s)
I20260809 06:50:53.112272 26377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/data/info.pb
I20260809 06:50:53.115386 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 26377
I20260809 06:50:53.115592 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-0/wal/instance
I20260809 06:50:53.116174 28487 external_mini_cluster.cc:1428] Setting key de34896def7e12d8ff4561b63af12738
I20260809 06:50:53.119081 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.194:0
--local_ip_for_outbound_sockets=127.27.209.194
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44439
--builtin_ntp_servers=127.27.209.212:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:50:53.124958 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:53.114726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.193:47465 (local address 127.27.209.254:44439)
0809 06:50:53.115182 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:53.115186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:53.116479 (+ 1293us) server_negotiation.cc:407] Connection header received
0809 06:50:53.118762 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:53.118768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:53.118848 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:53.118940 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:53.120105 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.120806 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.122284 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.122696 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.122804 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:53.124131 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:53.124136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:53.124139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:53.124156 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:53.124203 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:53.124215 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:53.124287 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:53.124514 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 06:50:53.124617 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":249,"threads_started":1}
I20260809 06:50:53.125861 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:53.115326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44439 (local address 127.27.209.193:47465)
0809 06:50:53.116316 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:53.116372 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:53.118508 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:53.119202 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:53.119215 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:53.119339 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:53.119939 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:53.119956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.120951 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.120955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:53.122146 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:53.122158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.122912 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.122919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:53.123143 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:53.123829 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:53.123832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:53.123869 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:53.123948 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:53.124303 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:53.124314 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:53.124322 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:50:53.124466 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 06:50:53.124837 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":327,"threads_started":1}
I20260809 06:50:53.127403 26579 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44439
I20260809 06:50:53.127745 26579 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:53.128616 26579 heartbeater.cc:511] Master 127.27.209.254:44439 requested a full tablet report, sending...
I20260809 06:50:53.130935 26299 ts_manager.cc:194] Registered new tserver with Master: 2dd56bebbdf245e4904bc62e121ea648 (127.27.209.193:38501)
I20260809 06:50:53.132427 26299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.193:47465
W20260809 06:50:53.452528 26583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:53.452805 26583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:53.452896 26583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:53.460068 26583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:53.460156 26583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:53.460265 26583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.194
I20260809 06:50:53.468925 26583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.209.194
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.194
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:53.470166 26583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:53.471972 26583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:53.484467 26589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:53.484508 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:53.484428 26591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:53.485499 26583 server_base.cc:1034] running on GCE node
I20260809 06:50:53.486083 26583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:53.487031 26583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:53.488234 26583 hybrid_clock.cc:648] HybridClock initialized: now 1786258253488184 us; error 39 us; skew 500 ppm
I20260809 06:50:53.488543 26583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:53.490674 26583 webserver.cc:460] Webserver started at http://127.27.209.194:46121/ using document root <none> and password file <none>
I20260809 06:50:53.491338 26583 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:53.491441 26583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:53.491781 26583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:53.494535 26583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data/instance:
uuid: "f810885343bc40b49dc5efb9b47c2d25"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "a9ae39b72202b6685bf3c34c23ad65d7"
server_key_iv: "836eee41a411cd607875257201391cc6"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.495293 26583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance:
uuid: "f810885343bc40b49dc5efb9b47c2d25"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "a9ae39b72202b6685bf3c34c23ad65d7"
server_key_iv: "836eee41a411cd607875257201391cc6"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.500120 26583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 06:50:53.503378 26603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.504686 26583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:53.504839 26583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal
uuid: "f810885343bc40b49dc5efb9b47c2d25"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "a9ae39b72202b6685bf3c34c23ad65d7"
server_key_iv: "836eee41a411cd607875257201391cc6"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.505030 26583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:53.518494 26583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:53.519218 26583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:53.519496 26583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:53.520426 26583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:53.521800 26583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:53.521896 26583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.522006 26583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:53.522078 26583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.552516 26583 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.194:42169
I20260809 06:50:53.552568 26778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.194:42169 every 8 connection(s)
I20260809 06:50:53.553890 26583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/data/info.pb
I20260809 06:50:53.559633 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 26583
I20260809 06:50:53.559824 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal/instance
I20260809 06:50:53.560389 28487 external_mini_cluster.cc:1428] Setting key 8384139d08289c4271d9e96609874ffd
I20260809 06:50:53.563019 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.209.195:0
--local_ip_for_outbound_sockets=127.27.209.195
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44439
--builtin_ntp_servers=127.27.209.212:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 06:50:53.566488 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:53.556348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:43787 (local address 127.27.209.254:44439)
0809 06:50:53.556569 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:53.556575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:53.558019 (+ 1444us) server_negotiation.cc:407] Connection header received
0809 06:50:53.559984 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:53.559991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:53.560077 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:53.560175 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:53.561426 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.562237 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.563918 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.564314 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.564425 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:53.565709 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:53.565714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:53.565717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:53.565734 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:53.565768 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:53.565779 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:53.565853 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:53.566065 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 06:50:53.566170 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:50:53.567462 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:53.556955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44439 (local address 127.27.209.194:43787)
0809 06:50:53.557861 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:53.557904 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:53.559637 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:53.560454 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:53.560465 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:53.560580 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:53.561275 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:53.561292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.562386 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.562392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:53.563782 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:53.563795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:53.564530 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:53.564539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:53.564755 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:53.565396 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:53.565399 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:53.565438 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:53.565513 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:53.565868 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:53.565880 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:53.565888 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 06:50:53.566040 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 06:50:53.566426 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":268,"threads_started":1}
I20260809 06:50:53.568708 26779 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44439
I20260809 06:50:53.569007 26779 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:53.569871 26779 heartbeater.cc:511] Master 127.27.209.254:44439 requested a full tablet report, sending...
I20260809 06:50:53.571332 26299 ts_manager.cc:194] Registered new tserver with Master: f810885343bc40b49dc5efb9b47c2d25 (127.27.209.194:42169)
I20260809 06:50:53.572324 26299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.194:43787
W20260809 06:50:53.893784 26783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:53.894047 26783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:53.894140 26783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:53.901252 26783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:53.901358 26783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:53.901473 26783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.209.195
I20260809 06:50:53.910657 26783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.209.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.209.195
--webserver_port=0
--tserver_master_addrs=127.27.209.254:44439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.209.195
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:53.911939 26783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:53.913736 26783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:53.926273 26793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:53.926311 26788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:53.926347 26791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:53.926994 26783 server_base.cc:1034] running on GCE node
I20260809 06:50:53.927608 26783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:53.928506 26783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:53.929699 26783 hybrid_clock.cc:648] HybridClock initialized: now 1786258253929651 us; error 44 us; skew 500 ppm
I20260809 06:50:53.930024 26783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 06:50:53.932266 26783 webserver.cc:460] Webserver started at http://127.27.209.195:45305/ using document root <none> and password file <none>
I20260809 06:50:53.932934 26783 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:53.933028 26783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:53.933350 26783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:53.936172 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data/instance:
uuid: "91fc75657df1400f9415588977ddd6b5"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "f7d243c975e59981b35241f154d524a3"
server_key_iv: "dd669c267bc645f1c3d08545d055339c"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.936933 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance:
uuid: "91fc75657df1400f9415588977ddd6b5"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "f7d243c975e59981b35241f154d524a3"
server_key_iv: "dd669c267bc645f1c3d08545d055339c"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.942063 26783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:50:53.945389 26802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.946626 26783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:50:53.946774 26783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal
uuid: "91fc75657df1400f9415588977ddd6b5"
format_stamp: "Formatted at 2026-08-09 06:50:53 on dist-test-slave-jpj1"
server_key: "f7d243c975e59981b35241f154d524a3"
server_key_iv: "dd669c267bc645f1c3d08545d055339c"
server_key_version: "encryptionkey@0"
I20260809 06:50:53.946969 26783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:53.962564 26783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:53.963320 26783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:53.963661 26783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:53.964578 26783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 06:50:53.965969 26783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 06:50:53.966060 26783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.966164 26783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 06:50:53.966241 26783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:53.995955 26783 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.195:37927
I20260809 06:50:53.996002 27002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.195:37927 every 8 connection(s)
I20260809 06:50:53.997236 26783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/data/info.pb
I20260809 06:50:54.002094 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 26783
I20260809 06:50:54.002282 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal/instance
I20260809 06:50:54.002835 28487 external_mini_cluster.cc:1428] Setting key ddf869e35fcfb3ab99786bdb7eff0e89
I20260809 06:50:54.010334 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:53.999715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.195:59657 (local address 127.27.209.254:44439)
0809 06:50:53.999889 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:53.999893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.001291 (+ 1398us) server_negotiation.cc:407] Connection header received
0809 06:50:54.003596 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.003601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.003680 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.003786 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:54.005125 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.006108 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.007649 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.008116 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.008242 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.009640 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:54.009645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:54.009649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:54.009668 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:54.009703 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:54.009715 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:54.009791 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.009989 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.010097 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:50:54.011332 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.000240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44439 (local address 127.27.209.195:59657)
0809 06:50:54.001125 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.001172 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:54.003301 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:54.004020 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:54.004032 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:54.004153 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:50:54.004959 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.004980 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.006256 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.006260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.007493 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.007506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.008346 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.008354 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.008567 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.009314 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 06:50:54.009317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:50:54.009358 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 06:50:54.009440 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:50:54.009805 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:50:54.009814 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:50:54.009820 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 06:50:54.009971 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 06:50:54.010288 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":260,"threads_started":1}
I20260809 06:50:54.012590 27003 heartbeater.cc:347] Connected to a master server at 127.27.209.254:44439
I20260809 06:50:54.012923 27003 heartbeater.cc:464] Registering TS with master...
I20260809 06:50:54.013679 27003 heartbeater.cc:511] Master 127.27.209.254:44439 requested a full tablet report, sending...
I20260809 06:50:54.015161 26299 ts_manager.cc:194] Registered new tserver with Master: 91fc75657df1400f9415588977ddd6b5 (127.27.209.195:37927)
I20260809 06:50:54.016029 26299 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.209.195:59657
I20260809 06:50:54.018236 28487 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 06:50:54.027901 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.022272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.27.209.254:44439)
0809 06:50:54.022432 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.022436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.022926 (+ 490us) server_negotiation.cc:407] Connection header received
0809 06:50:54.023056 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.023060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.023140 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.023221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:54.023715 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.024390 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.025663 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.026199 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.026320 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.027218 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:54.027222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:54.027225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:54.027243 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:54.027274 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:54.027286 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:54.027352 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.027580 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.027670 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:50:54.032410 26299 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 06:50:54.039703 26299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 06:50:54.062738 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.052175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36510)
0809 06:50:54.052999 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.053055 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:54.053216 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:54.055338 (+ 2122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:54.055347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:54.055385 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:54.055960 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.055973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.057766 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.057770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.060091 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.060110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.061766 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.061773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.062003 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.062067 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 06:50:54.062183 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 06:50:54.062376 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":432,"threads_started":1}
I20260809 06:50:54.062747 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.052772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.27.209.193:38501)
0809 06:50:54.054613 (+ 1841us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.054623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.054657 (+ 34us) server_negotiation.cc:407] Connection header received
0809 06:50:54.054776 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.054785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.055083 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.055284 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:54.056147 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.057589 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.060332 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.061569 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.061720 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.061944 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.062245 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.062417 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1670,"thread_start_us":1526,"threads_started":1}
I20260809 06:50:54.064735 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.052378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.194:42169 (local address 127.0.0.1:59636)
0809 06:50:54.053689 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.053710 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:54.053843 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:54.057352 (+ 3509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:54.057357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:54.057382 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:54.057711 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.057724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.059598 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.059604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.061768 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.061785 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.063952 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.063959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.064143 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.064177 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 06:50:54.064260 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 06:50:54.064397 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":1049,"threads_started":1}
I20260809 06:50:54.065325 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.053869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37927 (local address 127.0.0.1:56196)
0809 06:50:54.056154 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.056370 (+ 216us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:54.056489 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:54.058459 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:54.058464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:54.058487 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:54.058883 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.058894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.060632 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.060637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.062681 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.062697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.064581 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.064589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.064789 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.064821 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 06:50:54.064885 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 06:50:54.065004 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2158,"mutex_wait_us":1,"thread_start_us":1996,"threads_started":1}
I20260809 06:50:54.066577 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.054130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.27.209.195:37927)
0809 06:50:54.058043 (+ 3913us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.058049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.058074 (+ 25us) server_negotiation.cc:407] Connection header received
0809 06:50:54.058148 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.058153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.058292 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.058457 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:54.059066 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.060458 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.062955 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.064352 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.065629 (+ 1277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.065988 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.066074 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.066256 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3772,"thread_start_us":3637,"threads_started":1}
I20260809 06:50:54.067619 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.053271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.27.209.194:42169)
0809 06:50:54.056729 (+ 3458us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.056738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.056783 (+ 45us) server_negotiation.cc:407] Connection header received
0809 06:50:54.056890 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.056898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.057125 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.057373 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:54.057899 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.059422 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.061991 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.063752 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.063942 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.066973 (+ 3031us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.067058 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.067226 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":612,"threads_started":1}
I20260809 06:50:54.067440 26469 tablet_service.cc:1467] Processing CreateTablet for tablet f5b8729703634fc4a0a18dd34ceac0ae (DEFAULT_TABLE table=test-table [id=4cd8f374a03544cc91336da0a5283a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:54.069137 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b8729703634fc4a0a18dd34ceac0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:54.070312 26889 tablet_service.cc:1467] Processing CreateTablet for tablet f5b8729703634fc4a0a18dd34ceac0ae (DEFAULT_TABLE table=test-table [id=4cd8f374a03544cc91336da0a5283a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:54.071269 26677 tablet_service.cc:1467] Processing CreateTablet for tablet f5b8729703634fc4a0a18dd34ceac0ae (DEFAULT_TABLE table=test-table [id=4cd8f374a03544cc91336da0a5283a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 06:50:54.071838 26889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b8729703634fc4a0a18dd34ceac0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:54.072662 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5b8729703634fc4a0a18dd34ceac0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:54.080773 27034 tablet_bootstrap.cc:492] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Bootstrap starting.
I20260809 06:50:54.084503 27034 tablet_bootstrap.cc:654] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:54.085552 27034 log.cc:824] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:54.086208 27036 tablet_bootstrap.cc:492] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Bootstrap starting.
I20260809 06:50:54.087644 27037 tablet_bootstrap.cc:492] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Bootstrap starting.
I20260809 06:50:54.090103 27036 tablet_bootstrap.cc:654] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:54.090431 27037 tablet_bootstrap.cc:654] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:54.091475 27036 log.cc:824] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:54.091512 27037 log.cc:824] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:54.094732 27034 tablet_bootstrap.cc:492] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: No bootstrap required, opened a new log
I20260809 06:50:54.094987 27034 ts_tablet_manager.cc:1397] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260809 06:50:54.095031 27036 tablet_bootstrap.cc:492] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: No bootstrap required, opened a new log
I20260809 06:50:54.095289 27036 ts_tablet_manager.cc:1397] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 06:50:54.095336 27037 tablet_bootstrap.cc:492] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: No bootstrap required, opened a new log
I20260809 06:50:54.095599 27037 ts_tablet_manager.cc:1397] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260809 06:50:54.102157 27034 raft_consensus.cc:348] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.102680 27034 raft_consensus.cc:374] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:54.102814 27034 raft_consensus.cc:729] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd56bebbdf245e4904bc62e121ea648, State: Initialized, Role: FOLLOWER
I20260809 06:50:54.103521 27034 consensus_queue.cc:260] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.105242 27036 raft_consensus.cc:348] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.105719 27036 raft_consensus.cc:374] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:54.105736 26579 heartbeater.cc:503] Master 127.27.209.254:44439 was elected leader, sending a full tablet report...
I20260809 06:50:54.105863 27036 raft_consensus.cc:729] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fc75657df1400f9415588977ddd6b5, State: Initialized, Role: FOLLOWER
I20260809 06:50:54.106434 27034 ts_tablet_manager.cc:1428] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 06:50:54.106612 27036 consensus_queue.cc:260] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.106956 27037 raft_consensus.cc:348] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.107435 27037 raft_consensus.cc:374] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:54.107620 27037 raft_consensus.cc:729] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f810885343bc40b49dc5efb9b47c2d25, State: Initialized, Role: FOLLOWER
I20260809 06:50:54.107774 27003 heartbeater.cc:503] Master 127.27.209.254:44439 was elected leader, sending a full tablet report...
I20260809 06:50:54.108471 27036 ts_tablet_manager.cc:1428] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260809 06:50:54.108436 27037 consensus_queue.cc:260] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.110141 26779 heartbeater.cc:503] Master 127.27.209.254:44439 was elected leader, sending a full tablet report...
I20260809 06:50:54.110642 27037 ts_tablet_manager.cc:1428] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
W20260809 06:50:54.114398 26580 tablet.cc:2369] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:54.184755 27046 raft_consensus.cc:482] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 06:50:54.185055 27046 raft_consensus.cc:504] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.186895 27046 leader_election.cc:288] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dd56bebbdf245e4904bc62e121ea648 (127.27.209.193:38501), 91fc75657df1400f9415588977ddd6b5 (127.27.209.195:37927)
I20260809 06:50:54.193733 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.187480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:32893 (local address 127.27.209.195:37927)
0809 06:50:54.187706 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.187713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.188461 (+ 748us) server_negotiation.cc:407] Connection header received
0809 06:50:54.188621 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.188625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.188686 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.188760 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:54.189398 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.190120 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.192126 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.192888 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.193036 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.193103 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.193394 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.193511 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 06:50:54.193743 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.187431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.209.194:38435 (local address 127.27.209.193:38501)
0809 06:50:54.187678 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.187686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.188410 (+ 724us) server_negotiation.cc:407] Connection header received
0809 06:50:54.188594 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.188598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.188671 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.188754 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 06:50:54.189419 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.190112 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.192102 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.192856 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.192990 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.193060 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.193377 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.193494 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:50:54.193827 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.187015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.27.209.194:38435)
0809 06:50:54.188298 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.188314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:54.188456 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:54.188772 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:54.188776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:54.188799 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:54.189279 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.189290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.190238 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.190244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.191906 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.191918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.193037 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.193044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.193222 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.193251 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 06:50:54.193332 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 06:50:54.193452 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":101,"thread_start_us":952,"threads_started":1}
I20260809 06:50:54.193934 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.187113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.195:37927 (local address 127.27.209.194:32893)
0809 06:50:54.188349 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.188364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 06:50:54.188487 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:50:54.188795 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:50:54.188799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:50:54.188823 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 06:50:54.189232 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.189242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.190256 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.190261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.191908 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:50:54.191921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.193072 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.193078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:50:54.193236 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.193265 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 06:50:54.193343 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 06:50:54.193460 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"mutex_wait_us":73,"thread_start_us":769,"threads_started":1}
I20260809 06:50:54.194857 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b8729703634fc4a0a18dd34ceac0ae" candidate_uuid: "f810885343bc40b49dc5efb9b47c2d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd56bebbdf245e4904bc62e121ea648" is_pre_election: true
I20260809 06:50:54.194850 26920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b8729703634fc4a0a18dd34ceac0ae" candidate_uuid: "f810885343bc40b49dc5efb9b47c2d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fc75657df1400f9415588977ddd6b5" is_pre_election: true
I20260809 06:50:54.195319 26920 raft_consensus.cc:2454] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f810885343bc40b49dc5efb9b47c2d25 in term 0.
I20260809 06:50:54.195335 26499 raft_consensus.cc:2454] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f810885343bc40b49dc5efb9b47c2d25 in term 0.
I20260809 06:50:54.195958 26609 leader_election.cc:302] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91fc75657df1400f9415588977ddd6b5, f810885343bc40b49dc5efb9b47c2d25; no voters:
I20260809 06:50:54.196314 27046 raft_consensus.cc:2781] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 06:50:54.196461 27046 raft_consensus.cc:482] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 06:50:54.196564 27046 raft_consensus.cc:3037] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:54.198668 27046 raft_consensus.cc:504] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.199177 27046 leader_election.cc:288] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [CANDIDATE]: Term 1 election: Requested vote from peers 2dd56bebbdf245e4904bc62e121ea648 (127.27.209.193:38501), 91fc75657df1400f9415588977ddd6b5 (127.27.209.195:37927)
I20260809 06:50:54.199582 26499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b8729703634fc4a0a18dd34ceac0ae" candidate_uuid: "f810885343bc40b49dc5efb9b47c2d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd56bebbdf245e4904bc62e121ea648"
I20260809 06:50:54.199649 26920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5b8729703634fc4a0a18dd34ceac0ae" candidate_uuid: "f810885343bc40b49dc5efb9b47c2d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91fc75657df1400f9415588977ddd6b5"
I20260809 06:50:54.199760 26499 raft_consensus.cc:3037] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:54.199824 26920 raft_consensus.cc:3037] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:54.201911 26920 raft_consensus.cc:2454] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f810885343bc40b49dc5efb9b47c2d25 in term 1.
I20260809 06:50:54.201915 26499 raft_consensus.cc:2454] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f810885343bc40b49dc5efb9b47c2d25 in term 1.
I20260809 06:50:54.202273 26607 leader_election.cc:302] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2dd56bebbdf245e4904bc62e121ea648, 91fc75657df1400f9415588977ddd6b5, f810885343bc40b49dc5efb9b47c2d25; no voters:
I20260809 06:50:54.202597 27046 raft_consensus.cc:2781] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:54.202991 27046 raft_consensus.cc:686] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 1 LEADER]: Becoming Leader. State: Replica: f810885343bc40b49dc5efb9b47c2d25, State: Running, Role: LEADER
I20260809 06:50:54.203486 27046 consensus_queue.cc:237] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } }
I20260809 06:50:54.208061 26297 catalog_manager.cc:5495] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 reported cstate change: term changed from 0 to 1, leader changed from <none> to f810885343bc40b49dc5efb9b47c2d25 (127.27.209.194). New cstate: current_term: 1 leader_uuid: "f810885343bc40b49dc5efb9b47c2d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f810885343bc40b49dc5efb9b47c2d25" member_type: VOTER last_known_addr { host: "127.27.209.194" port: 42169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 } health_report { overall_health: UNKNOWN } } }
W20260809 06:50:54.248847 27007 tablet.cc:2369] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:50:54.254421 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.249127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.27.209.194:42169)
0809 06:50:54.249335 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.249341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.249426 (+ 85us) server_negotiation.cc:407] Connection header received
0809 06:50:54.249648 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.249652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.249733 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.249916 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:50:54.250208 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.250937 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.252453 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.252855 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.252947 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.253334 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:50:54.253859 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:50:54.253982 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.254075 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.254155 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:50:54.262917 26499 raft_consensus.cc:1261] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 1 FOLLOWER]: Refusing update from remote peer f810885343bc40b49dc5efb9b47c2d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:54.263000 26920 raft_consensus.cc:1261] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 1 FOLLOWER]: Refusing update from remote peer f810885343bc40b49dc5efb9b47c2d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:50:54.264004 27049 consensus_queue.cc:1035] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91fc75657df1400f9415588977ddd6b5" member_type: VOTER last_known_addr { host: "127.27.209.195" port: 37927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:54.264328 27046 consensus_queue.cc:1035] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd56bebbdf245e4904bc62e121ea648" member_type: VOTER last_known_addr { host: "127.27.209.193" port: 38501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:50:54.275148 27052 mvcc.cc:205] Tried to move back new op lower bound from 7316513809450356736 to 7316513809221607424. Current Snapshot: MvccSnapshot[applied={T|T < 7316513809450356736}]
I20260809 06:50:54.290160 26299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:54.290428 26299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 06:50:54.293668 26299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 292d77ebe0584725bbe9854e30c0b3c2: Sending DeleteTablet for 3 replicas of tablet f5b8729703634fc4a0a18dd34ceac0ae
I20260809 06:50:54.294755 26469 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b8729703634fc4a0a18dd34ceac0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:54 UTC) from {username='slave'} at 127.0.0.1:36510
I20260809 06:50:54.294839 26677 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b8729703634fc4a0a18dd34ceac0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:54 UTC) from {username='slave'} at 127.0.0.1:59636
I20260809 06:50:54.296123 26889 tablet_service.cc:1514] Processing DeleteTablet for tablet f5b8729703634fc4a0a18dd34ceac0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 06:50:54 UTC) from {username='slave'} at 127.0.0.1:56196
I20260809 06:50:54.297350 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 26377
I20260809 06:50:54.311224 27067 tablet_replica.cc:330] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: stopping tablet replica
I20260809 06:50:54.311846 27068 tablet_replica.cc:330] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: stopping tablet replica
I20260809 06:50:54.312319 27068 raft_consensus.cc:2229] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:54.312851 27068 raft_consensus.cc:2258] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:54.313635 27067 raft_consensus.cc:2229] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 06:50:54.314291 27067 raft_consensus.cc:2258] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:54.315258 27068 ts_tablet_manager.cc:1905] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:54.318667 27067 ts_tablet_manager.cc:1905] T f5b8729703634fc4a0a18dd34ceac0ae P 2dd56bebbdf245e4904bc62e121ea648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:54.321309 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.298662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.27.209.193:38501)
0809 06:50:54.298836 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.298842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.303919 (+ 5077us) server_negotiation.cc:407] Connection header received
0809 06:50:54.304804 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.304810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.304943 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.305020 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:54.307327 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.308536 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.314716 (+ 6180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.315448 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.315575 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.319486 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:54.319499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:54.319518 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:54.320330 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:54.320400 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:54.320429 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:54.320517 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.320779 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.320915 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 06:50:54.322691 26568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:54.322993 27068 ts_tablet_manager.cc:1918] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:54.323239 27068 log.cc:1196] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-2/wal/wals/f5b8729703634fc4a0a18dd34ceac0ae
I20260809 06:50:54.323472 27065 tablet_replica.cc:330] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: stopping tablet replica
I20260809 06:50:54.323906 27068 ts_tablet_manager.cc:1939] T f5b8729703634fc4a0a18dd34ceac0ae P 91fc75657df1400f9415588977ddd6b5: Deleting consensus metadata
I20260809 06:50:54.324077 27065 raft_consensus.cc:2229] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 1 LEADER]: Raft consensus shutting down.
I20260809 06:50:54.324954 27065 raft_consensus.cc:2258] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 06:50:54.326162 26273 catalog_manager.cc:4841] TS 91fc75657df1400f9415588977ddd6b5 (127.27.209.195:37927): tablet f5b8729703634fc4a0a18dd34ceac0ae (table test-table [id=4cd8f374a03544cc91336da0a5283a58]) successfully deleted
I20260809 06:50:54.327450 27065 ts_tablet_manager.cc:1905] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 06:50:54.335186 27065 ts_tablet_manager.cc:1918] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 06:50:54.335359 27065 log.cc:1196] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Deleting WAL directory at /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786257935933888-28487-0/minicluster-data/ts-1/wal/wals/f5b8729703634fc4a0a18dd34ceac0ae
I20260809 06:50:54.335855 27065 ts_tablet_manager.cc:1939] T f5b8729703634fc4a0a18dd34ceac0ae P f810885343bc40b49dc5efb9b47c2d25: Deleting consensus metadata
I20260809 06:50:54.337392 26273 catalog_manager.cc:4841] TS f810885343bc40b49dc5efb9b47c2d25 (127.27.209.194:42169): tablet f5b8729703634fc4a0a18dd34ceac0ae (table test-table [id=4cd8f374a03544cc91336da0a5283a58]) successfully deleted
I20260809 06:50:54.431308 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 26377
I20260809 06:50:54.451359 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 26583
W20260809 06:50:54.451411 26271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.209.193:38501 (error 108)
I20260809 06:50:54.453492 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.452941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36532)
0809 06:50:54.453181 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.453319 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260809 06:50:54.453739 26271 catalog_manager.cc:4584] TS 2dd56bebbdf245e4904bc62e121ea648 (127.27.209.193:38501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5b8729703634fc4a0a18dd34ceac0ae: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
I20260809 06:50:54.458961 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.452499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.27.209.194:42169)
0809 06:50:54.452674 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.452680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.452701 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:50:54.452779 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.452784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.452865 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.452966 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:54.453390 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.454300 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.455635 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.456316 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.456446 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.457401 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:54.457412 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:54.457435 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:54.458209 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:54.458277 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:54.458304 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:54.458411 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.458494 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.458594 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:50:54.459710 26771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:54.507329 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.506744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36544)
0809 06:50:54.507002 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.507119 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 06:50:54.508458 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.508069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36556)
0809 06:50:54.508210 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.508297 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 06:50:54.569854 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.569310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36568)
0809 06:50:54.569528 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.569647 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:50:54.569950 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 26583
I20260809 06:50:54.577318 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.576889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36580)
0809 06:50:54.577063 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.577165 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 06:50:54.593787 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 26783
I20260809 06:50:54.600519 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.594550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.27.209.195:37927)
0809 06:50:54.594725 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:54.594730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:54.594770 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:50:54.594948 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:54.594952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:54.595046 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:54.595133 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:54.595589 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.596342 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.597614 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:54.598017 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:54.598130 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:54.599140 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:54.599150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:54.599165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:54.599813 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:54.599865 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:54.599897 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:54.599999 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:54.600134 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 06:50:54.600241 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:50:54.601152 26995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:54.657896 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.657415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36584)
0809 06:50:54.657609 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.657697 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 06:50:54.658914 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:54.658519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.193:38501 (local address 127.0.0.1:36596)
0809 06:50:54.658725 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 06:50:54.658798 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.193:38501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 06:50:54.689349 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 26783
I20260809 06:50:54.710124 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 26245
I20260809 06:50:54.711176 26343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:54.817915 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 26245
2026-08-09T06:50:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2754 ms)
[----------] 6 tests from EncryptionPolicyTest (22382 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:50:54 dist-test-slave-jpj1 krb5kdc[27081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:50:54 dist-test-slave-jpj1 krb5kdc[27081](info): set up 2 sockets
Aug 09 06:50:54 dist-test-slave-jpj1 krb5kdc[27081](info): commencing operation
krb5kdc: starting...
W20260809 06:50:56.035041 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:50:56 dist-test-slave-jpj1 krb5kdc[27081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:50:56.326373 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:56.658237 27098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:56.658519 27098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:56.658612 27098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:56.665899 27098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:56.665997 27098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:56.666049 27098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:56.666115 27098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:56.666158 27098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:56.675549 27098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:56.676723 27098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:56.677953 27098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:56.689071 27108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:56.689107 27111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:56.689147 27109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:56.690317 27098 server_base.cc:1034] running on GCE node
I20260809 06:50:56.691093 27098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:56.692502 27098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:56.693710 27098 hybrid_clock.cc:648] HybridClock initialized: now 1786258256693655 us; error 36 us; skew 500 ppm
I20260809 06:50:56.694156 27098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:56 dist-test-slave-jpj1 krb5kdc[27081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:56.699891 27098 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:50:56.701337 27098 webserver.cc:460] Webserver started at http://127.0.0.1:39987/ using document root <none> and password file <none>
I20260809 06:50:56.702026 27098 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:56.702145 27098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:56.702466 27098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:56.705513 27098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917"
format_stamp: "Formatted at 2026-08-09 06:50:56 on dist-test-slave-jpj1"
server_key: "5ebb1126bf3c5b296dc6b35396fa5c65"
server_key_iv: "83c98dadeb50f8579f9e352906dd291c"
server_key_version: "encryptionkey@0"
I20260809 06:50:56.706284 27098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917"
format_stamp: "Formatted at 2026-08-09 06:50:56 on dist-test-slave-jpj1"
server_key: "5ebb1126bf3c5b296dc6b35396fa5c65"
server_key_iv: "83c98dadeb50f8579f9e352906dd291c"
server_key_version: "encryptionkey@0"
I20260809 06:50:56.711416 27098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:50:56.714787 27121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:56.716130 27098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:50:56.716303 27098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917"
format_stamp: "Formatted at 2026-08-09 06:50:56 on dist-test-slave-jpj1"
server_key: "5ebb1126bf3c5b296dc6b35396fa5c65"
server_key_iv: "83c98dadeb50f8579f9e352906dd291c"
server_key_version: "encryptionkey@0"
I20260809 06:50:56.716483 27098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:56.746536 27098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:56.750595 27098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:56.750870 27098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:56.768419 27098 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42229
I20260809 06:50:56.768455 27206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42229 every 8 connection(s)
I20260809 06:50:56.769769 27098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:56.773387 27211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:56.773881 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 27098
I20260809 06:50:56.774091 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:56.774933 28487 external_mini_cluster.cc:1428] Setting key 74913b0c9516710347ec9979bcd0764f
I20260809 06:50:56.786952 27211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917: Bootstrap starting.
Aug 09 06:50:56 dist-test-slave-jpj1 krb5kdc[27081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:50:56.790664 27211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:56.791837 27211 log.cc:824] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:56.794795 27211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917: No bootstrap required, opened a new log
I20260809 06:50:56.798583 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:56.777042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46457 (local address 127.0.0.1:42229)
0809 06:50:56.777785 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:56.777796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:56.777838 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:50:56.779664 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:56.779689 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:56.780082 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:56.780398 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:56.782203 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:56.783591 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:56.785436 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:56.786081 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:56.786206 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:56.791061 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:56.791088 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:56.791106 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:56.791142 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:56.794326 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:56.795212 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:56.795218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:56.795224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:56.795412 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:56.795935 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:56.795938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:56.795940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:56.796472 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:56.796734 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:56.797274 (+ 540us) server_negotiation.cc:299] Negotiation successful
0809 06:50:56.797558 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":334,"threads_started":1}
I20260809 06:50:56.802421 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:50:56.803313 27211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260809 06:50:56.803736 27211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:56.803828 27211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f9ee2ec9c94d728cd3e6b5b5d42917, State: Initialized, Role: FOLLOWER
I20260809 06:50:56.804415 27211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260809 06:50:56.804684 27211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:56.804829 27211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:56.805007 27211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:56.808014 27211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260809 06:50:56.808525 27211 leader_election.cc:302] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f9ee2ec9c94d728cd3e6b5b5d42917; no voters:
I20260809 06:50:56.809206 27211 leader_election.cc:288] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:56.809363 27233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:56.810364 27233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [term 1 LEADER]: Becoming Leader. State: Replica: 76f9ee2ec9c94d728cd3e6b5b5d42917, State: Running, Role: LEADER
I20260809 06:50:56.811004 27233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260809 06:50:56.811375 27211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:56.813747 27236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f9ee2ec9c94d728cd3e6b5b5d42917. Latest consensus state: current_term: 1 leader_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } } }
I20260809 06:50:56.814177 27236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:56.814627 27234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f9ee2ec9c94d728cd3e6b5b5d42917" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } } }
I20260809 06:50:56.814927 27234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:56.815114 27240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:56.823896 27240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:56.825469 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:56.807960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59481 (local address 127.0.0.1:42229)
0809 06:50:56.808137 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:56.808144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:56.808564 (+ 420us) server_negotiation.cc:407] Connection header received
0809 06:50:56.808722 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:56.808727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:56.808801 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:56.808877 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:56.810782 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:56.811952 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:56.813708 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:56.814199 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:56.814321 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:56.817624 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:56.817632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:56.817638 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:50:56.817679 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:56.821688 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:56.822703 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:56.822709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:56.822712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:56.822858 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:50:56.823465 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:50:56.823470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:50:56.823473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:50:56.823944 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:56.824199 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:56.824923 (+ 724us) server_negotiation.cc:299] Negotiation successful
0809 06:50:56.825110 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:50:56.835777 27240 catalog_manager.cc:1347] Generated new cluster ID: 7d1cba15443743fb887d93408f982bf2
I20260809 06:50:56.835883 27240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:56.852609 27240 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:56.854450 27240 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:56.863101 27240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f9ee2ec9c94d728cd3e6b5b5d42917: Generated new TSK 0
I20260809 06:50:56.864118 27240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:50:56.876468 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27098
I20260809 06:50:56.878484 27201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:50:56.968182 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 27098
W20260809 06:50:56.990967 27079 connection.cc:535] client connection to 127.0.0.1:42229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42229): Connection reset by peer (error 104)
2026-08-09T06:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2145 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:50:57 dist-test-slave-jpj1 krb5kdc[27266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:50:57 dist-test-slave-jpj1 krb5kdc[27266](info): set up 2 sockets
Aug 09 06:50:57 dist-test-slave-jpj1 krb5kdc[27266](info): commencing operation
krb5kdc: starting...
W20260809 06:50:58.202181 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.005s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:50:58 dist-test-slave-jpj1 krb5kdc[27266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:50:58.482096 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:50:58.811509 27281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:50:58.811806 27281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:50:58.811934 27281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:50:58.818974 27281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:50:58.819078 27281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:50:58.819154 27281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:50:58.819231 27281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:50:58.819295 27281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:50:58.828841 27281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:50:58.830083 27281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:50:58.831375 27281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:50:58.842639 27294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:58.842707 27291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:50:58.842685 27292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:50:58.843432 27281 server_base.cc:1034] running on GCE node
I20260809 06:50:58.844240 27281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:50:58.845670 27281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:50:58.846877 27281 hybrid_clock.cc:648] HybridClock initialized: now 1786258258846835 us; error 31 us; skew 500 ppm
I20260809 06:50:58.847211 27281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:50:58 dist-test-slave-jpj1 krb5kdc[27266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:50:58.850411 27281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:50:58.851791 27281 webserver.cc:460] Webserver started at http://127.0.0.1:39481/ using document root <none> and password file <none>
I20260809 06:50:58.852413 27281 fs_manager.cc:359] Metadata directory not provided
I20260809 06:50:58.852510 27281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:50:58.852762 27281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:50:58.855700 27281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "5a23a65e52034b83a33a7e422101a980"
format_stamp: "Formatted at 2026-08-09 06:50:58 on dist-test-slave-jpj1"
server_key: "c5d4904131b9498105542068b420aa85"
server_key_iv: "a1fbe59e434fc03734ea3d20620e2eec"
server_key_version: "encryptionkey@0"
I20260809 06:50:58.856451 27281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "5a23a65e52034b83a33a7e422101a980"
format_stamp: "Formatted at 2026-08-09 06:50:58 on dist-test-slave-jpj1"
server_key: "c5d4904131b9498105542068b420aa85"
server_key_iv: "a1fbe59e434fc03734ea3d20620e2eec"
server_key_version: "encryptionkey@0"
I20260809 06:50:58.861307 27281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:50:58.864539 27308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:50:58.865782 27281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:50:58.865942 27281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "5a23a65e52034b83a33a7e422101a980"
format_stamp: "Formatted at 2026-08-09 06:50:58 on dist-test-slave-jpj1"
server_key: "c5d4904131b9498105542068b420aa85"
server_key_iv: "a1fbe59e434fc03734ea3d20620e2eec"
server_key_version: "encryptionkey@0"
I20260809 06:50:58.866110 27281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:50:58.879391 27281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:50:58.883441 27281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:50:58.883769 27281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:50:58.900063 27281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33877
I20260809 06:50:58.900110 27390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33877 every 8 connection(s)
I20260809 06:50:58.901384 27281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:50:58.904989 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:50:58.906635 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 27281
I20260809 06:50:58.906786 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:50:58.907197 28487 external_mini_cluster.cc:1428] Setting key effeba6b1b9363ab2f7e0a429e0a80af
I20260809 06:50:58.918232 27394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980: Bootstrap starting.
I20260809 06:50:58.919946 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:58.908871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39967 (local address 127.0.0.1:33877)
0809 06:50:58.909650 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:58.909664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:58.909711 (+ 47us) server_negotiation.cc:407] Connection header received
0809 06:50:58.911249 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:58.911274 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:58.911689 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:58.912006 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:58.912557 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:58.913793 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:58.915653 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:58.916336 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:58.916461 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:58.917712 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:58.917732 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:58.917745 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:58.918256 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:58.918327 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:58.918351 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:58.918485 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:58.918758 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 06:50:58.918929 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":322,"threads_started":1}
I20260809 06:50:58.921752 27394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980: Neither blocks nor log segments found. Creating new log.
I20260809 06:50:58.922917 27394 log.cc:824] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980: Log is configured to *not* fsync() on all Append() calls
I20260809 06:50:58.923727 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:50:58.925637 27394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980: No bootstrap required, opened a new log
I20260809 06:50:58.934979 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:50:58.928273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46649 (local address 127.0.0.1:33877)
0809 06:50:58.928479 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 06:50:58.928488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 06:50:58.928785 (+ 297us) server_negotiation.cc:407] Connection header received
0809 06:50:58.928954 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:50:58.928961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:50:58.929028 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:50:58.929120 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:50:58.929685 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:58.930487 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:58.932440 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:50:58.932894 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:50:58.932999 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:50:58.934240 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:50:58.934245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:50:58.934255 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:50:58.934278 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:50:58.934335 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:50:58.934351 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:50:58.934432 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:50:58.934612 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 06:50:58.934660 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 06:50:58.935356 27394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a23a65e52034b83a33a7e422101a980" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260809 06:50:58.935874 27394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:50:58.935985 27394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a23a65e52034b83a33a7e422101a980, State: Initialized, Role: FOLLOWER
I20260809 06:50:58.936652 27394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a23a65e52034b83a33a7e422101a980" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260809 06:50:58.936878 27394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:50:58.937016 27394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:50:58.937170 27394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:50:58.939337 27394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a23a65e52034b83a33a7e422101a980" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260809 06:50:58.939853 27394 leader_election.cc:302] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a23a65e52034b83a33a7e422101a980; no voters:
I20260809 06:50:58.940394 27394 leader_election.cc:288] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:50:58.940514 27415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:50:58.941175 27415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [term 1 LEADER]: Becoming Leader. State: Replica: 5a23a65e52034b83a33a7e422101a980, State: Running, Role: LEADER
I20260809 06:50:58.941732 27415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a23a65e52034b83a33a7e422101a980" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } }
I20260809 06:50:58.942595 27394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:50:58.944866 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a23a65e52034b83a33a7e422101a980. Latest consensus state: current_term: 1 leader_uuid: "5a23a65e52034b83a33a7e422101a980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a23a65e52034b83a33a7e422101a980" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } } }
I20260809 06:50:58.945290 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:58.945578 27416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a23a65e52034b83a33a7e422101a980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a23a65e52034b83a33a7e422101a980" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33877 } } }
I20260809 06:50:58.945921 27416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980 [sys.catalog]: This master's current role is: LEADER
I20260809 06:50:58.947273 27432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:50:58.952596 27432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:50:58.961462 27432 catalog_manager.cc:1347] Generated new cluster ID: 785204310e1744a4ab545322a12804a1
I20260809 06:50:58.961581 27432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:50:59.004472 27432 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:50:59.005803 27432 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:50:59.019598 27432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a23a65e52034b83a33a7e422101a980: Generated new TSK 0
I20260809 06:50:59.020479 27432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:50:59.035894 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27281
W20260809 06:50:59.037191 27384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39967
W20260809 06:50:59.037988 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27281: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:50:59.038187 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 27281
W20260809 06:50:59.063771 27263 connection.cc:535] client connection to 127.0.0.1:33877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33877): Connection reset by peer (error 104)
2026-08-09T06:50:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:50:59 dist-test-slave-jpj1 krb5kdc[27446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:50:59 dist-test-slave-jpj1 krb5kdc[27446](info): set up 2 sockets
Aug 09 06:50:59 dist-test-slave-jpj1 krb5kdc[27446](info): commencing operation
krb5kdc: starting...
W20260809 06:51:00.256057 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:00 dist-test-slave-jpj1 krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:00.547286 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:00.885145 27473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:00.885437 27473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:00.885535 27473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:00.892475 27473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:00.892582 27473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:00.892637 27473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:00.892702 27473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:00.892745 27473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:00.902352 27473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:00.903626 27473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:00.904888 27473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:00.916051 27478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:00.916029 27479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:00.916316 27482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:00.917253 27473 server_base.cc:1034] running on GCE node
I20260809 06:51:00.917984 27473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:00.919378 27473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:00.920605 27473 hybrid_clock.cc:648] HybridClock initialized: now 1786258260920545 us; error 54 us; skew 500 ppm
I20260809 06:51:00.920948 27473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:00 dist-test-slave-jpj1 krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:00.924132 27473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:00.925487 27473 webserver.cc:460] Webserver started at http://127.0.0.1:42019/ using document root <none> and password file <none>
I20260809 06:51:00.926139 27473 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:00.926251 27473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:00.926533 27473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:00.929454 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "c24d3e4d48bd4c10bf370cef7f37cacd"
format_stamp: "Formatted at 2026-08-09 06:51:00 on dist-test-slave-jpj1"
server_key: "8f6d5b326f729d2acf6f848862d5add6"
server_key_iv: "8d2691abf2356756f6aeb7325e30c830"
server_key_version: "encryptionkey@0"
I20260809 06:51:00.930220 27473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "c24d3e4d48bd4c10bf370cef7f37cacd"
format_stamp: "Formatted at 2026-08-09 06:51:00 on dist-test-slave-jpj1"
server_key: "8f6d5b326f729d2acf6f848862d5add6"
server_key_iv: "8d2691abf2356756f6aeb7325e30c830"
server_key_version: "encryptionkey@0"
I20260809 06:51:00.935194 27473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:51:00.938491 27495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:00.939751 27473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:00.939913 27473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "c24d3e4d48bd4c10bf370cef7f37cacd"
format_stamp: "Formatted at 2026-08-09 06:51:00 on dist-test-slave-jpj1"
server_key: "8f6d5b326f729d2acf6f848862d5add6"
server_key_iv: "8d2691abf2356756f6aeb7325e30c830"
server_key_version: "encryptionkey@0"
I20260809 06:51:00.940085 27473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:00.945499 27473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:00.949630 27473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:00.949900 27473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:00.966729 27473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36077
I20260809 06:51:00.966774 27574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36077 every 8 connection(s)
I20260809 06:51:00.968101 27473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:00.972015 27577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:00.975193 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 27473
I20260809 06:51:00.975383 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:00.975960 28487 external_mini_cluster.cc:1428] Setting key a54771184558b700e545aea248ff87fc
I20260809 06:51:00.984007 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:00.978287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36155 (local address 127.0.0.1:36077)
0809 06:51:00.979083 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:00.979095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:00.979135 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:51:00.980803 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:00.980827 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:00.981165 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:00.981522 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:00.982484 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:00.982491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:00.982503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:00.982518 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:00.982575 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:00.982594 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:00.982792 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:00.982922 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 06:51:00.983094 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":333,"threads_started":1}
I20260809 06:51:00.984378 27577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd: Bootstrap starting.
I20260809 06:51:00.987725 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:00.987996 27577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:00.988976 27577 log.cc:824] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:00.991948 27577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd: No bootstrap required, opened a new log
I20260809 06:51:00.994141 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:00.991844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44837 (local address 127.0.0.1:36077)
0809 06:51:00.992000 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:00.992005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:00.992376 (+ 371us) server_negotiation.cc:407] Connection header received
0809 06:51:00.992522 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:00.992530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:00.992573 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:00.992650 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:00.993518 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:00.993527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:00.993531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:00.993549 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:00.993584 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:00.993599 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:00.993712 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:00.993817 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 06:51:00.993877 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 06:51:00.999634 27577 raft_consensus.cc:348] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260809 06:51:00.999991 27577 raft_consensus.cc:374] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:01.000089 27577 raft_consensus.cc:729] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24d3e4d48bd4c10bf370cef7f37cacd, State: Initialized, Role: FOLLOWER
I20260809 06:51:01.000613 27577 consensus_queue.cc:260] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260809 06:51:01.000841 27577 raft_consensus.cc:388] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:01.000975 27577 raft_consensus.cc:482] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:01.001163 27577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:01.003327 27577 raft_consensus.cc:504] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260809 06:51:01.003823 27577 leader_election.cc:302] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c24d3e4d48bd4c10bf370cef7f37cacd; no voters:
I20260809 06:51:01.004369 27577 leader_election.cc:288] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:01.004500 27595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:01.005160 27595 raft_consensus.cc:686] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [term 1 LEADER]: Becoming Leader. State: Replica: c24d3e4d48bd4c10bf370cef7f37cacd, State: Running, Role: LEADER
I20260809 06:51:01.005686 27595 consensus_queue.cc:237] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260809 06:51:01.006363 27577 sys_catalog.cc:564] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:01.009020 27596 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } } }
I20260809 06:51:01.008927 27598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c24d3e4d48bd4c10bf370cef7f37cacd. Latest consensus state: current_term: 1 leader_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24d3e4d48bd4c10bf370cef7f37cacd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } } }
I20260809 06:51:01.012943 27596 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:01.012967 27598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:01.016433 27612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:01.020690 27612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:01.029348 27612 catalog_manager.cc:1347] Generated new cluster ID: 1f02c5ab7c3d4007830955fe3fecc2a7
I20260809 06:51:01.029461 27612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:01.047231 27612 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:01.047798 27612 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:01.061447 27612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c24d3e4d48bd4c10bf370cef7f37cacd: Generated new TSK 0
I20260809 06:51:01.062500 27612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:01.066421 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27473
W20260809 06:51:01.068058 27566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36155
W20260809 06:51:01.068698 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:01.068878 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 27473
2026-08-09T06:51:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:01 dist-test-slave-jpj1 krb5kdc[27626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:01 dist-test-slave-jpj1 krb5kdc[27626](info): set up 2 sockets
Aug 09 06:51:01 dist-test-slave-jpj1 krb5kdc[27626](info): commencing operation
krb5kdc: starting...
W20260809 06:51:02.286418 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:02 dist-test-slave-jpj1 krb5kdc[27626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:02.574894 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:02.901731 27650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:02.902048 27650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:02.902185 27650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:02.909440 27650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:02.909559 27650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:02.909649 27650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:02.909740 27650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:02.909799 27650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:02.919697 27650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:02.921016 27650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:02.922317 27650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:02.933970 27660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:02.933935 27662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:02.933935 27665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:02.934597 27650 server_base.cc:1034] running on GCE node
I20260809 06:51:02.935390 27650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:02.936794 27650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:02.937991 27650 hybrid_clock.cc:648] HybridClock initialized: now 1786258262937945 us; error 41 us; skew 500 ppm
I20260809 06:51:02.938369 27650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:02 dist-test-slave-jpj1 krb5kdc[27626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:02.944335 27650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:02.945770 27650 webserver.cc:460] Webserver started at http://127.0.0.1:46683/ using document root <none> and password file <none>
I20260809 06:51:02.946435 27650 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:02.946557 27650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:02.946849 27650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:02.949863 27650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "ce788d03cc7d40e88dbb9c6b985565a7"
format_stamp: "Formatted at 2026-08-09 06:51:02 on dist-test-slave-jpj1"
server_key: "b1e1b2b913d21fdc775612671d9bc351"
server_key_iv: "f6c0eba0c9c869fe8046a2cf116aaddd"
server_key_version: "encryptionkey@0"
I20260809 06:51:02.950628 27650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "ce788d03cc7d40e88dbb9c6b985565a7"
format_stamp: "Formatted at 2026-08-09 06:51:02 on dist-test-slave-jpj1"
server_key: "b1e1b2b913d21fdc775612671d9bc351"
server_key_iv: "f6c0eba0c9c869fe8046a2cf116aaddd"
server_key_version: "encryptionkey@0"
I20260809 06:51:02.955790 27650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:51:02.959295 27681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:02.960680 27650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:02.960847 27650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "ce788d03cc7d40e88dbb9c6b985565a7"
format_stamp: "Formatted at 2026-08-09 06:51:02 on dist-test-slave-jpj1"
server_key: "b1e1b2b913d21fdc775612671d9bc351"
server_key_iv: "f6c0eba0c9c869fe8046a2cf116aaddd"
server_key_version: "encryptionkey@0"
I20260809 06:51:02.961030 27650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:02.977085 27650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:02.981184 27650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:02.981449 27650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:02.998459 27650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46779
I20260809 06:51:02.998482 27768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46779 every 8 connection(s)
I20260809 06:51:02.999759 27650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:03.003366 27769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:03.009802 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 27650
I20260809 06:51:03.009958 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:03.010390 28487 external_mini_cluster.cc:1428] Setting key 9bcb989339f835f65d7c384d37b1e97b
I20260809 06:51:03.014370 27769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7: Bootstrap starting.
I20260809 06:51:03.017521 27769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:03.018563 27769 log.cc:824] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:03.021464 27769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7: No bootstrap required, opened a new log
Aug 09 06:51:03 dist-test-slave-jpj1 krb5kdc[27626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:51:03.029505 27769 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260809 06:51:03.029922 27769 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:03.030019 27769 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce788d03cc7d40e88dbb9c6b985565a7, State: Initialized, Role: FOLLOWER
I20260809 06:51:03.030664 27769 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260809 06:51:03.030926 27769 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:03.031080 27769 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:03.031265 27769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:03.032459 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:03.012216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51091 (local address 127.0.0.1:46779)
0809 06:51:03.013039 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:03.013051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:03.013092 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:51:03.014780 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:03.014803 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:03.015181 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:03.015567 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:03.017043 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:03.018389 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:03.020017 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:03.020676 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:03.020842 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:03.025033 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:03.025041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:03.025056 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:03.025092 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:03.028245 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:03.028947 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:03.028953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:03.028960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:03.029127 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:03.029515 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:03.029519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:03.029521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:03.030112 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:03.030422 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:03.030870 (+ 448us) server_negotiation.cc:299] Negotiation successful
0809 06:51:03.031333 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":354,"threads_started":1}
I20260809 06:51:03.034140 27769 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260809 06:51:03.034637 27769 leader_election.cc:302] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce788d03cc7d40e88dbb9c6b985565a7; no voters:
I20260809 06:51:03.035348 27769 leader_election.cc:288] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:03.035468 27779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:03.036262 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:03.037590 27769 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:03.042326 27779 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [term 1 LEADER]: Becoming Leader. State: Replica: ce788d03cc7d40e88dbb9c6b985565a7, State: Running, Role: LEADER
I20260809 06:51:03.042954 27779 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } }
I20260809 06:51:03.046205 27782 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } } }
I20260809 06:51:03.046578 27782 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:03.047484 27783 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce788d03cc7d40e88dbb9c6b985565a7. Latest consensus state: current_term: 1 leader_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce788d03cc7d40e88dbb9c6b985565a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46779 } } }
I20260809 06:51:03.047858 27783 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:03.055056 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:03.041048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55725 (local address 127.0.0.1:46779)
0809 06:51:03.041205 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:03.041210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:03.041226 (+ 16us) server_negotiation.cc:407] Connection header received
0809 06:51:03.041264 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:03.041267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:03.041317 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:03.041393 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:03.042989 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:03.043889 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:03.045287 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:03.046202 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:03.046351 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:03.049622 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:03.049629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:03.049635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:03.049677 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:03.052559 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:03.053162 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:03.053167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:03.053171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:03.053300 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:03.053643 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:03.053647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:03.053649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:03.053965 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:03.054173 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:03.054516 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 06:51:03.054713 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 06:51:03.056844 27802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:03.061702 27802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:03.071472 27802 catalog_manager.cc:1347] Generated new cluster ID: 9db29f2c89cf410cb949368b16cafa35
I20260809 06:51:03.071604 27802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:03.087700 27802 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:03.089565 27802 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:03.101222 27802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce788d03cc7d40e88dbb9c6b985565a7: Generated new TSK 0
I20260809 06:51:03.102284 27802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:03.116644 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27650
I20260809 06:51:03.118069 27763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:03.210551 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 27650
2026-08-09T06:51:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:03 dist-test-slave-jpj1 krb5kdc[27826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:03 dist-test-slave-jpj1 krb5kdc[27826](info): set up 2 sockets
Aug 09 06:51:03 dist-test-slave-jpj1 krb5kdc[27826](info): commencing operation
krb5kdc: starting...
W20260809 06:51:04.413890 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:04 dist-test-slave-jpj1 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:04.695295 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:05.024341 27845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:05.024675 27845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:05.024827 27845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:05.032086 27845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:05.032213 27845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:05.032298 27845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:05.032378 27845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:05.032440 27845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:05.042215 27845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:05.043514 27845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:05.044839 27845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:05.056452 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:05.056419 27856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:05.056378 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:05.057307 27845 server_base.cc:1034] running on GCE node
I20260809 06:51:05.058054 27845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:05.059381 27845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:05.060580 27845 hybrid_clock.cc:648] HybridClock initialized: now 1786258265060523 us; error 42 us; skew 500 ppm
I20260809 06:51:05.060915 27845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:05 dist-test-slave-jpj1 krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:05.063907 27845 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:05.065301 27845 webserver.cc:460] Webserver started at http://127.0.0.1:44977/ using document root <none> and password file <none>
I20260809 06:51:05.065953 27845 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:05.066072 27845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:05.066370 27845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:05.069357 27845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "29db39321caf40909dfe312153167073"
format_stamp: "Formatted at 2026-08-09 06:51:05 on dist-test-slave-jpj1"
server_key: "3a04dcdff9d8f75b2caa3f95cbe4f72f"
server_key_iv: "5bfa6cc8a882786a737a2a4916cffd4f"
server_key_version: "encryptionkey@0"
I20260809 06:51:05.070116 27845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "29db39321caf40909dfe312153167073"
format_stamp: "Formatted at 2026-08-09 06:51:05 on dist-test-slave-jpj1"
server_key: "3a04dcdff9d8f75b2caa3f95cbe4f72f"
server_key_iv: "5bfa6cc8a882786a737a2a4916cffd4f"
server_key_version: "encryptionkey@0"
I20260809 06:51:05.075320 27845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 06:51:05.078648 27866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:05.079968 27845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:05.080137 27845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "29db39321caf40909dfe312153167073"
format_stamp: "Formatted at 2026-08-09 06:51:05 on dist-test-slave-jpj1"
server_key: "3a04dcdff9d8f75b2caa3f95cbe4f72f"
server_key_iv: "5bfa6cc8a882786a737a2a4916cffd4f"
server_key_version: "encryptionkey@0"
I20260809 06:51:05.080319 27845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:05.094762 27845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:05.098474 27845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:05.098702 27845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:05.115908 27845 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42241
I20260809 06:51:05.115938 27940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42241 every 8 connection(s)
I20260809 06:51:05.117215 27845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:05.120816 27941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:05.121484 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 27845
I20260809 06:51:05.121675 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:05.122134 28487 external_mini_cluster.cc:1428] Setting key 102ef6f5d3f2dd71068015bfe1cedd05
I20260809 06:51:05.133737 27941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073: Bootstrap starting.
I20260809 06:51:05.135268 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:05.123964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38155 (local address 127.0.0.1:42241)
0809 06:51:05.124684 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:05.124695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:05.124742 (+ 47us) server_negotiation.cc:407] Connection header received
0809 06:51:05.126540 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:05.126563 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:05.126942 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:05.127257 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:05.127851 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:05.129068 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:05.130916 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:05.131652 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:05.131793 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:05.132961 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:05.132968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:05.132983 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:05.133519 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:05.133595 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:05.133619 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:05.133724 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:05.133997 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 06:51:05.134273 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":326,"threads_started":1}
I20260809 06:51:05.137665 27941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:05.138980 27941 log.cc:824] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:05.140302 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:05.142247 27941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073: No bootstrap required, opened a new log
I20260809 06:51:05.152002 27944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:05.145471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50903 (local address 127.0.0.1:42241)
0809 06:51:05.145671 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:05.145679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:05.146142 (+ 463us) server_negotiation.cc:407] Connection header received
0809 06:51:05.146336 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:05.146340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:05.146401 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:05.146488 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:05.147058 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:05.147842 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:05.149646 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:05.150116 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:05.150206 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:05.151303 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:05.151308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:05.151311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:05.151327 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:05.151363 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:05.151379 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:05.151442 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:05.151630 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 06:51:05.151700 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:51:05.152309 27941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29db39321caf40909dfe312153167073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260809 06:51:05.152732 27941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:05.152835 27941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29db39321caf40909dfe312153167073, State: Initialized, Role: FOLLOWER
I20260809 06:51:05.153481 27941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29db39321caf40909dfe312153167073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260809 06:51:05.153702 27941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:05.153836 27941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:05.153980 27941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:05.156199 27941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29db39321caf40909dfe312153167073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260809 06:51:05.156690 27941 leader_election.cc:302] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29db39321caf40909dfe312153167073; no voters:
I20260809 06:51:05.157281 27941 leader_election.cc:288] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:05.157418 27956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:05.158066 27956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [term 1 LEADER]: Becoming Leader. State: Replica: 29db39321caf40909dfe312153167073, State: Running, Role: LEADER
I20260809 06:51:05.158563 27956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29db39321caf40909dfe312153167073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } }
I20260809 06:51:05.159322 27941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:05.161367 27959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29db39321caf40909dfe312153167073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29db39321caf40909dfe312153167073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } } }
I20260809 06:51:05.161741 27959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:05.162163 27960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29db39321caf40909dfe312153167073. Latest consensus state: current_term: 1 leader_uuid: "29db39321caf40909dfe312153167073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29db39321caf40909dfe312153167073" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42241 } } }
I20260809 06:51:05.162488 27960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:05.163861 27971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:05.168972 27971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:05.177440 27971 catalog_manager.cc:1347] Generated new cluster ID: 3d16d98216254d1b995352f170a4fee7
I20260809 06:51:05.177551 27971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:05.196161 27971 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:05.197425 27971 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:05.208920 27971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29db39321caf40909dfe312153167073: Generated new TSK 0
I20260809 06:51:05.209707 27971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:05.219359 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27845
W20260809 06:51:05.220359 27934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38155
W20260809 06:51:05.221010 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 27845: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:05.221138 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 27845
2026-08-09T06:51:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:05 dist-test-slave-jpj1 krb5kdc[27990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:05 dist-test-slave-jpj1 krb5kdc[27990](info): set up 2 sockets
Aug 09 06:51:05 dist-test-slave-jpj1 krb5kdc[27990](info): commencing operation
krb5kdc: starting...
W20260809 06:51:06.441718 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:06 dist-test-slave-jpj1 krb5kdc[27990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:06.732920 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:07.067970 28010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:07.068257 28010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:07.068392 28010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:07.075587 28010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:07.075716 28010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:07.075796 28010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:07.075874 28010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:07.075932 28010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:07.085788 28010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:07.087139 28010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:07.088470 28010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:07.100150 28018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:07.100178 28019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:07.100116 28021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:07.101147 28010 server_base.cc:1034] running on GCE node
I20260809 06:51:07.101899 28010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:07.103261 28010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:07.104473 28010 hybrid_clock.cc:648] HybridClock initialized: now 1786258267104420 us; error 43 us; skew 500 ppm
I20260809 06:51:07.104815 28010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:07 dist-test-slave-jpj1 krb5kdc[27990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:07.107882 28010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:07.109299 28010 webserver.cc:460] Webserver started at http://127.0.0.1:33743/ using document root <none> and password file <none>
I20260809 06:51:07.109967 28010 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:07.110070 28010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:07.110332 28010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:07.113351 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab"
format_stamp: "Formatted at 2026-08-09 06:51:07 on dist-test-slave-jpj1"
server_key: "3ba11c8a6b8c99a743168b39d598cf32"
server_key_iv: "ad9f2fe1829a0139cf9fc9bb8b5e9da0"
server_key_version: "encryptionkey@0"
I20260809 06:51:07.114080 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab"
format_stamp: "Formatted at 2026-08-09 06:51:07 on dist-test-slave-jpj1"
server_key: "3ba11c8a6b8c99a743168b39d598cf32"
server_key_iv: "ad9f2fe1829a0139cf9fc9bb8b5e9da0"
server_key_version: "encryptionkey@0"
I20260809 06:51:07.119092 28010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:51:07.122532 28028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:07.123852 28010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 06:51:07.124010 28010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab"
format_stamp: "Formatted at 2026-08-09 06:51:07 on dist-test-slave-jpj1"
server_key: "3ba11c8a6b8c99a743168b39d598cf32"
server_key_iv: "ad9f2fe1829a0139cf9fc9bb8b5e9da0"
server_key_version: "encryptionkey@0"
I20260809 06:51:07.124203 28010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:07.129987 28010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:07.133917 28010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:07.134199 28010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:07.152379 28010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43009
I20260809 06:51:07.152419 28099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43009 every 8 connection(s)
I20260809 06:51:07.153786 28010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:07.157572 28100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:07.160535 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28010
I20260809 06:51:07.160717 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:07.161274 28487 external_mini_cluster.cc:1428] Setting key 118b36a041a6b38d693ca113ffb2e518
I20260809 06:51:07.169175 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:07.163166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38507 (local address 127.0.0.1:43009)
0809 06:51:07.164006 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:07.164019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:07.164059 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:51:07.165713 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:07.165737 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:07.166084 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:07.166435 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:07.167505 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:07.167511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:07.167525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:07.167571 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:07.167650 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:07.167681 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:07.167866 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:07.167962 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 06:51:07.168164 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":383,"threads_started":1}
I20260809 06:51:07.169914 28100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab: Bootstrap starting.
I20260809 06:51:07.173069 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:07.173646 28100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:07.174903 28100 log.cc:824] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:07.177856 28100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab: No bootstrap required, opened a new log
I20260809 06:51:07.180473 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:07.178057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49465 (local address 127.0.0.1:43009)
0809 06:51:07.178261 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:07.178266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:07.178625 (+ 359us) server_negotiation.cc:407] Connection header received
0809 06:51:07.178812 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:07.178815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:07.178861 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:07.178937 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:07.179877 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:07.179881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:07.179884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:07.179902 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:07.179942 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:07.179959 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:07.180050 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:07.180150 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 06:51:07.180199 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 06:51:07.185896 28100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260809 06:51:07.186286 28100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:07.186379 28100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824e9cb05f4d4d838b4f0a8640d2b4ab, State: Initialized, Role: FOLLOWER
I20260809 06:51:07.186981 28100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260809 06:51:07.187215 28100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:07.187353 28100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:07.187527 28100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:07.189764 28100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260809 06:51:07.190264 28100 leader_election.cc:302] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824e9cb05f4d4d838b4f0a8640d2b4ab; no voters:
I20260809 06:51:07.190893 28100 leader_election.cc:288] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:07.191041 28119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:07.191753 28119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [term 1 LEADER]: Becoming Leader. State: Replica: 824e9cb05f4d4d838b4f0a8640d2b4ab, State: Running, Role: LEADER
I20260809 06:51:07.192265 28119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260809 06:51:07.193025 28100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:07.195055 28120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } } }
I20260809 06:51:07.195066 28122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824e9cb05f4d4d838b4f0a8640d2b4ab. Latest consensus state: current_term: 1 leader_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824e9cb05f4d4d838b4f0a8640d2b4ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } } }
I20260809 06:51:07.195566 28120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:07.195575 28122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:07.197608 28128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:07.202936 28128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:07.212297 28128 catalog_manager.cc:1347] Generated new cluster ID: db70dfcf1ba740888050c5c45bafbfce
I20260809 06:51:07.212409 28128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:07.233156 28128 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:07.233603 28128 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:07.243587 28128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824e9cb05f4d4d838b4f0a8640d2b4ab: Generated new TSK 0
I20260809 06:51:07.244439 28128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:07.252990 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28010
W20260809 06:51:07.254235 28094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38507
W20260809 06:51:07.255105 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28010: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:07.255285 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28010
2026-08-09T06:51:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:07 dist-test-slave-jpj1 krb5kdc[28152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:07 dist-test-slave-jpj1 krb5kdc[28152](info): set up 2 sockets
Aug 09 06:51:07 dist-test-slave-jpj1 krb5kdc[28152](info): commencing operation
krb5kdc: starting...
W20260809 06:51:08.484407 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:08 dist-test-slave-jpj1 krb5kdc[28152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:08.768301 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:09.105770 28172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:09.106051 28172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:09.106261 28172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:09.113476 28172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:09.113592 28172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:09.113653 28172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:09.113714 28172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:09.113763 28172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:09.123245 28172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:09.124429 28172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:09.125689 28172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:09.137138 28179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:09.137108 28188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:09.137198 28181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:09.144992 28172 server_base.cc:1034] running on GCE node
I20260809 06:51:09.145773 28172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:09.147149 28172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:09.148346 28172 hybrid_clock.cc:648] HybridClock initialized: now 1786258269148287 us; error 50 us; skew 500 ppm
I20260809 06:51:09.148684 28172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:09 dist-test-slave-jpj1 krb5kdc[28152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:09.152016 28172 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:09.153842 28172 webserver.cc:460] Webserver started at http://127.0.0.1:36641/ using document root <none> and password file <none>
I20260809 06:51:09.154527 28172 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:09.154639 28172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:09.154953 28172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:09.157992 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "e6974415deb54e4ab646edb37fc7a3a4"
format_stamp: "Formatted at 2026-08-09 06:51:09 on dist-test-slave-jpj1"
server_key: "090e13de7aeb8a6fda18fbb1665994ef"
server_key_iv: "ad561aa0377f06001d5074d08e3034c4"
server_key_version: "encryptionkey@0"
I20260809 06:51:09.158766 28172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "e6974415deb54e4ab646edb37fc7a3a4"
format_stamp: "Formatted at 2026-08-09 06:51:09 on dist-test-slave-jpj1"
server_key: "090e13de7aeb8a6fda18fbb1665994ef"
server_key_iv: "ad561aa0377f06001d5074d08e3034c4"
server_key_version: "encryptionkey@0"
I20260809 06:51:09.163884 28172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 06:51:09.167212 28198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:09.168572 28172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:09.168751 28172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "e6974415deb54e4ab646edb37fc7a3a4"
format_stamp: "Formatted at 2026-08-09 06:51:09 on dist-test-slave-jpj1"
server_key: "090e13de7aeb8a6fda18fbb1665994ef"
server_key_iv: "ad561aa0377f06001d5074d08e3034c4"
server_key_version: "encryptionkey@0"
I20260809 06:51:09.168923 28172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:09.190025 28172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:09.193930 28172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:09.194195 28172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:09.212121 28172 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38967
I20260809 06:51:09.212180 28271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38967 every 8 connection(s)
I20260809 06:51:09.213452 28172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:09.214847 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28172
I20260809 06:51:09.215039 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:09.215659 28487 external_mini_cluster.cc:1428] Setting key 232439f450c1a045f032d19b4c73bec5
I20260809 06:51:09.218070 28272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:51:09 dist-test-slave-jpj1 krb5kdc[28152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:51:09.230965 28272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4: Bootstrap starting.
I20260809 06:51:09.235026 28272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:09.236287 28272 log.cc:824] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:09.239636 28272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4: No bootstrap required, opened a new log
I20260809 06:51:09.239898 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:09.217529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:56491 (local address 127.0.0.1:38967)
0809 06:51:09.218330 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:09.218342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:09.218384 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:51:09.220116 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:09.220139 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:09.220546 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:09.220872 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:09.222664 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:09.224001 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:09.225828 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:09.226513 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:09.226646 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:09.231495 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:09.231505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:09.231524 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:09.231607 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:09.235398 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:09.236241 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:09.236250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:09.236263 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:09.236477 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:09.236956 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:09.236961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:09.236963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:09.237511 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:09.237824 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:09.238238 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 06:51:09.238567 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":361,"threads_started":1}
I20260809 06:51:09.243808 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:09.247661 28272 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6974415deb54e4ab646edb37fc7a3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260809 06:51:09.248150 28272 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:09.248281 28272 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6974415deb54e4ab646edb37fc7a3a4, State: Initialized, Role: FOLLOWER
I20260809 06:51:09.248929 28272 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6974415deb54e4ab646edb37fc7a3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260809 06:51:09.249176 28272 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:09.249303 28272 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:09.249470 28272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:09.251747 28272 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6974415deb54e4ab646edb37fc7a3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260809 06:51:09.252229 28272 leader_election.cc:302] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6974415deb54e4ab646edb37fc7a3a4; no voters:
I20260809 06:51:09.252840 28272 leader_election.cc:288] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:09.252975 28289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:09.254041 28289 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [term 1 LEADER]: Becoming Leader. State: Replica: e6974415deb54e4ab646edb37fc7a3a4, State: Running, Role: LEADER
I20260809 06:51:09.254628 28289 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6974415deb54e4ab646edb37fc7a3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260809 06:51:09.255157 28272 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:09.257733 28291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6974415deb54e4ab646edb37fc7a3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6974415deb54e4ab646edb37fc7a3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } } }
I20260809 06:51:09.258126 28291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:09.258713 28292 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6974415deb54e4ab646edb37fc7a3a4. Latest consensus state: current_term: 1 leader_uuid: "e6974415deb54e4ab646edb37fc7a3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6974415deb54e4ab646edb37fc7a3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } } }
I20260809 06:51:09.259045 28292 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:09.260370 28295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:09.264802 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:09.248052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:45701 (local address 127.0.0.1:38967)
0809 06:51:09.248234 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:09.248239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:09.248836 (+ 597us) server_negotiation.cc:407] Connection header received
0809 06:51:09.249018 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:09.249024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:09.249090 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:09.249175 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:09.250782 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:09.251596 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:09.252921 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:09.253630 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:09.254361 (+ 731us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:09.256459 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:09.256465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:09.256470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:09.256515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:09.260950 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:09.261690 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:09.261694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:09.261697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:09.261849 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:09.262301 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:09.262305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:09.262307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:09.262724 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:09.263996 (+ 1272us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:09.264200 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 06:51:09.264385 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":81}
I20260809 06:51:09.266171 28295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:09.280200 28295 catalog_manager.cc:1347] Generated new cluster ID: 4fc570eb4e874884b9c15649ff5c60ed
I20260809 06:51:09.280346 28295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:09.295022 28295 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:09.296381 28295 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:09.307149 28295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6974415deb54e4ab646edb37fc7a3a4: Generated new TSK 0
I20260809 06:51:09.308104 28295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:09.317648 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28172
I20260809 06:51:09.319262 28263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:09.411394 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28172
W20260809 06:51:09.433705 28150 connection.cc:535] client connection to 127.0.0.1:38967 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38967): Connection reset by peer (error 104)
2026-08-09T06:51:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2153 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:09 dist-test-slave-jpj1 krb5kdc[28325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:09 dist-test-slave-jpj1 krb5kdc[28325](info): set up 2 sockets
Aug 09 06:51:09 dist-test-slave-jpj1 krb5kdc[28325](info): commencing operation
krb5kdc: starting...
W20260809 06:51:10.618094 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:10 dist-test-slave-jpj1 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:10.896505 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:11.228914 28343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:11.229224 28343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:11.229362 28343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:11.236526 28343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:11.236649 28343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:11.236728 28343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:11.236814 28343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:11.236876 28343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:11.246778 28343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:11.248124 28343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:11.249462 28343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:11.260828 28353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:11.260833 28355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:11.260833 28352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:11.261868 28343 server_base.cc:1034] running on GCE node
I20260809 06:51:11.262651 28343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:11.264082 28343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:11.265270 28343 hybrid_clock.cc:648] HybridClock initialized: now 1786258271265223 us; error 47 us; skew 500 ppm
I20260809 06:51:11.265592 28343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:11 dist-test-slave-jpj1 krb5kdc[28325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:11.268666 28343 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:11.270021 28343 webserver.cc:460] Webserver started at http://127.0.0.1:45631/ using document root <none> and password file <none>
I20260809 06:51:11.270646 28343 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:11.270746 28343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:11.270992 28343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:11.273999 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "40b31eeff56541529f87f099c3c7fac5"
format_stamp: "Formatted at 2026-08-09 06:51:11 on dist-test-slave-jpj1"
server_key: "b26fe0138dba326289d6b3cc37abe49d"
server_key_iv: "989794cd626927103d6e37ffd08b2baa"
server_key_version: "encryptionkey@0"
I20260809 06:51:11.274717 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "40b31eeff56541529f87f099c3c7fac5"
format_stamp: "Formatted at 2026-08-09 06:51:11 on dist-test-slave-jpj1"
server_key: "b26fe0138dba326289d6b3cc37abe49d"
server_key_iv: "989794cd626927103d6e37ffd08b2baa"
server_key_version: "encryptionkey@0"
I20260809 06:51:11.279721 28343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:51:11.283243 28369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:11.284632 28343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:11.284817 28343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "40b31eeff56541529f87f099c3c7fac5"
format_stamp: "Formatted at 2026-08-09 06:51:11 on dist-test-slave-jpj1"
server_key: "b26fe0138dba326289d6b3cc37abe49d"
server_key_iv: "989794cd626927103d6e37ffd08b2baa"
server_key_version: "encryptionkey@0"
I20260809 06:51:11.284996 28343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:11.309676 28343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:11.313583 28343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:11.313853 28343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:11.330531 28343 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41573
I20260809 06:51:11.330555 28450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41573 every 8 connection(s)
I20260809 06:51:11.331897 28343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:11.333806 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28343
I20260809 06:51:11.333971 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:11.334393 28487 external_mini_cluster.cc:1428] Setting key 9845ca39a7901848a3fc99e61d81ceb7
I20260809 06:51:11.335933 28451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:11.348621 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:11.336219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:56411 (local address 127.0.0.1:41573)
0809 06:51:11.336992 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:11.337007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:11.337056 (+ 49us) server_negotiation.cc:407] Connection header received
0809 06:51:11.339017 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:11.339045 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:11.339563 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:11.339994 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:11.340473 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:11.342427 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:11.343718 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:11.344697 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:11.344889 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:11.345951 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:11.345962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:11.345985 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:11.346679 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:11.346781 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:11.346816 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:11.346985 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:11.347198 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 06:51:11.347375 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":297,"threads_started":1}
I20260809 06:51:11.350221 28451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5: Bootstrap starting.
I20260809 06:51:11.352546 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:11.353416 28451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:11.354650 28451 log.cc:824] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:11.357604 28451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5: No bootstrap required, opened a new log
I20260809 06:51:11.363577 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:11.357520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:35695 (local address 127.0.0.1:41573)
0809 06:51:11.357660 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:11.357664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:11.358154 (+ 490us) server_negotiation.cc:407] Connection header received
0809 06:51:11.358331 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:11.358335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:11.358393 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:11.358475 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:11.358957 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:11.359829 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:11.361227 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:11.361651 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:11.361773 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:11.362855 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:11.362860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:11.362866 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:11.362882 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:11.362921 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:11.362935 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:11.363040 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:11.363244 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 06:51:11.363299 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 06:51:11.365865 28451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b31eeff56541529f87f099c3c7fac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41573 } }
I20260809 06:51:11.366211 28451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:11.366291 28451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b31eeff56541529f87f099c3c7fac5, State: Initialized, Role: FOLLOWER
I20260809 06:51:11.366811 28451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b31eeff56541529f87f099c3c7fac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41573 } }
I20260809 06:51:11.367012 28451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:11.367107 28451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:11.367234 28451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:11.369307 28451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b31eeff56541529f87f099c3c7fac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41573 } }
I20260809 06:51:11.369761 28451 leader_election.cc:302] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b31eeff56541529f87f099c3c7fac5; no voters:
I20260809 06:51:11.370289 28451 leader_election.cc:288] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:11.370435 28468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:11.371632 28468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [term 1 LEADER]: Becoming Leader. State: Replica: 40b31eeff56541529f87f099c3c7fac5, State: Running, Role: LEADER
I20260809 06:51:11.372160 28468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b31eeff56541529f87f099c3c7fac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41573 } }
I20260809 06:51:11.372561 28451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:11.374683 28470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b31eeff56541529f87f099c3c7fac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b31eeff56541529f87f099c3c7fac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41573 } } }
I20260809 06:51:11.375066 28470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:11.376030 28471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b31eeff56541529f87f099c3c7fac5. Latest consensus state: current_term: 1 leader_uuid: "40b31eeff56541529f87f099c3c7fac5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b31eeff56541529f87f099c3c7fac5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41573 } } }
I20260809 06:51:11.376394 28471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:11.377504 28480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:11.382637 28480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:11.391110 28480 catalog_manager.cc:1347] Generated new cluster ID: 32feb481e25c44f49efdfd00591cd936
I20260809 06:51:11.391213 28480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:11.401198 28480 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:11.402614 28480 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:11.413118 28480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b31eeff56541529f87f099c3c7fac5: Generated new TSK 0
I20260809 06:51:11.413903 28480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:11.420773 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28343
W20260809 06:51:11.422216 28439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:56411
W20260809 06:51:11.422892 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:11.423041 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28343
W20260809 06:51:11.446480 28321 connection.cc:535] client connection to 127.0.0.1:41573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41573): Connection reset by peer (error 104)
2026-08-09T06:51:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:11 dist-test-slave-jpj1 krb5kdc[28512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:11 dist-test-slave-jpj1 krb5kdc[28512](info): set up 2 sockets
Aug 09 06:51:11 dist-test-slave-jpj1 krb5kdc[28512](info): commencing operation
krb5kdc: starting...
W20260809 06:51:12.638104 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:12 dist-test-slave-jpj1 krb5kdc[28512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:12.921103 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:13.258189 28534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:13.258460 28534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:13.258591 28534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:13.265619 28534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:13.265730 28534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:13.265801 28534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:13.265890 28534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:13.265945 28534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:13.275979 28534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:13.277244 28534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:13.278636 28534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:13.289887 28546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:13.289844 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:13.289912 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:13.290560 28534 server_base.cc:1034] running on GCE node
I20260809 06:51:13.291363 28534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:13.292707 28534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:13.293929 28534 hybrid_clock.cc:648] HybridClock initialized: now 1786258273293880 us; error 37 us; skew 500 ppm
I20260809 06:51:13.294274 28534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:13 dist-test-slave-jpj1 krb5kdc[28512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:13.300401 28534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:13.301851 28534 webserver.cc:460] Webserver started at http://127.0.0.1:37661/ using document root <none> and password file <none>
I20260809 06:51:13.302491 28534 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:13.302604 28534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:13.302928 28534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:13.306172 28534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "a0557d3311cd4e2daf107b40ddb1d070"
format_stamp: "Formatted at 2026-08-09 06:51:13 on dist-test-slave-jpj1"
server_key: "38127fc613c454777ef45c6f648c25f5"
server_key_iv: "6861210e8354f7dc218b4f9360f2bbd4"
server_key_version: "encryptionkey@0"
I20260809 06:51:13.306950 28534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "a0557d3311cd4e2daf107b40ddb1d070"
format_stamp: "Formatted at 2026-08-09 06:51:13 on dist-test-slave-jpj1"
server_key: "38127fc613c454777ef45c6f648c25f5"
server_key_iv: "6861210e8354f7dc218b4f9360f2bbd4"
server_key_version: "encryptionkey@0"
I20260809 06:51:13.312093 28534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:51:13.315466 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:13.316756 28534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:51:13.316910 28534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "a0557d3311cd4e2daf107b40ddb1d070"
format_stamp: "Formatted at 2026-08-09 06:51:13 on dist-test-slave-jpj1"
server_key: "38127fc613c454777ef45c6f648c25f5"
server_key_iv: "6861210e8354f7dc218b4f9360f2bbd4"
server_key_version: "encryptionkey@0"
I20260809 06:51:13.317083 28534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:13.322862 28534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:13.326957 28534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:13.327236 28534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:13.345185 28534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36365
I20260809 06:51:13.345235 28639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36365 every 8 connection(s)
I20260809 06:51:13.346534 28534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:13.349725 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28534
I20260809 06:51:13.349902 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:13.350454 28487 external_mini_cluster.cc:1428] Setting key 123855ec39ee7e5d54de76454ea60fdf
I20260809 06:51:13.350751 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:13.358457 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:13.352284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:41307 (local address 127.0.0.1:36365)
0809 06:51:13.353017 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:13.353028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:13.353070 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:51:13.355241 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:13.355264 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:13.355620 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:13.355945 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:13.356967 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:13.356973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:13.356986 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:13.357003 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:13.357062 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:13.357081 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:13.357263 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:13.357368 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 06:51:13.357554 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":339,"threads_started":1}
I20260809 06:51:13.361848 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:13.364192 28640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070: Bootstrap starting.
I20260809 06:51:13.368079 28640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:13.369359 28640 log.cc:824] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:13.369407 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:13.367011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:53705 (local address 127.0.0.1:36365)
0809 06:51:13.367200 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:13.367206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:13.367558 (+ 352us) server_negotiation.cc:407] Connection header received
0809 06:51:13.367737 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:13.367741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:13.367787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:13.367870 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:13.368780 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:13.368784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:13.368787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:13.368801 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:13.368850 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:13.368865 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:13.368947 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:13.369091 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 06:51:13.369141 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 06:51:13.372087 28640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070: No bootstrap required, opened a new log
I20260809 06:51:13.379072 28640 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0557d3311cd4e2daf107b40ddb1d070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260809 06:51:13.379403 28640 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:13.379490 28640 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0557d3311cd4e2daf107b40ddb1d070, State: Initialized, Role: FOLLOWER
I20260809 06:51:13.380039 28640 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0557d3311cd4e2daf107b40ddb1d070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260809 06:51:13.380244 28640 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:13.380383 28640 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:13.380563 28640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:13.382666 28640 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0557d3311cd4e2daf107b40ddb1d070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260809 06:51:13.383105 28640 leader_election.cc:302] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0557d3311cd4e2daf107b40ddb1d070; no voters:
I20260809 06:51:13.383642 28640 leader_election.cc:288] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:13.383762 28657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:13.384737 28657 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [term 1 LEADER]: Becoming Leader. State: Replica: a0557d3311cd4e2daf107b40ddb1d070, State: Running, Role: LEADER
I20260809 06:51:13.385298 28657 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0557d3311cd4e2daf107b40ddb1d070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } }
I20260809 06:51:13.385650 28640 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:13.388064 28662 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0557d3311cd4e2daf107b40ddb1d070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0557d3311cd4e2daf107b40ddb1d070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } } }
I20260809 06:51:13.388484 28662 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:13.388906 28663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0557d3311cd4e2daf107b40ddb1d070. Latest consensus state: current_term: 1 leader_uuid: "a0557d3311cd4e2daf107b40ddb1d070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0557d3311cd4e2daf107b40ddb1d070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36365 } } }
I20260809 06:51:13.389258 28663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:13.390219 28674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:13.395217 28674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:13.407518 28674 catalog_manager.cc:1347] Generated new cluster ID: 3f15900a69134d11bd37dfd921a041cc
I20260809 06:51:13.407685 28674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:13.424757 28674 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:13.425149 28674 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:13.430823 28674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0557d3311cd4e2daf107b40ddb1d070: Generated new TSK 0
I20260809 06:51:13.431565 28674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:13.436408 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28534
W20260809 06:51:13.437348 28634 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:41307
W20260809 06:51:13.437968 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:13.438123 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28534
2026-08-09T06:51:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:13 dist-test-slave-jpj1 krb5kdc[28688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:13 dist-test-slave-jpj1 krb5kdc[28688](info): set up 2 sockets
Aug 09 06:51:13 dist-test-slave-jpj1 krb5kdc[28688](info): commencing operation
krb5kdc: starting...
W20260809 06:51:14.653118 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:14 dist-test-slave-jpj1 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:14.938045 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:15.273309 28704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:15.273579 28704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:15.273717 28704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:15.280794 28704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:15.280905 28704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:15.280957 28704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:15.281018 28704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:15.281061 28704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:15.290668 28704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:15.291893 28704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:15.293097 28704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:15.304316 28722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:15.304379 28712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:15.304356 28715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:15.305327 28704 server_base.cc:1034] running on GCE node
I20260809 06:51:15.306070 28704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:15.307366 28704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:15.308563 28704 hybrid_clock.cc:648] HybridClock initialized: now 1786258275308516 us; error 35 us; skew 500 ppm
I20260809 06:51:15.308894 28704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:15 dist-test-slave-jpj1 krb5kdc[28688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:15.311903 28704 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:15.313305 28704 webserver.cc:460] Webserver started at http://127.0.0.1:37789/ using document root <none> and password file <none>
I20260809 06:51:15.313951 28704 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:15.314069 28704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:15.314396 28704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:15.317338 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "f11f1cae693b4479b87689e7b8c146a1"
format_stamp: "Formatted at 2026-08-09 06:51:15 on dist-test-slave-jpj1"
server_key: "25dd893d9b910d18cdb52f00d4e97801"
server_key_iv: "5dd138b77c068502820889485cb0d21c"
server_key_version: "encryptionkey@0"
I20260809 06:51:15.318091 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "f11f1cae693b4479b87689e7b8c146a1"
format_stamp: "Formatted at 2026-08-09 06:51:15 on dist-test-slave-jpj1"
server_key: "25dd893d9b910d18cdb52f00d4e97801"
server_key_iv: "5dd138b77c068502820889485cb0d21c"
server_key_version: "encryptionkey@0"
I20260809 06:51:15.322986 28704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 06:51:15.326440 28735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:15.327780 28704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:15.327942 28704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "f11f1cae693b4479b87689e7b8c146a1"
format_stamp: "Formatted at 2026-08-09 06:51:15 on dist-test-slave-jpj1"
server_key: "25dd893d9b910d18cdb52f00d4e97801"
server_key_iv: "5dd138b77c068502820889485cb0d21c"
server_key_version: "encryptionkey@0"
I20260809 06:51:15.328114 28704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:15.350219 28704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:15.354451 28704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:15.354729 28704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:15.372149 28704 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40453
I20260809 06:51:15.372212 28819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40453 every 8 connection(s)
I20260809 06:51:15.373454 28704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:15.375432 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28704
I20260809 06:51:15.375620 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:15.376057 28487 external_mini_cluster.cc:1428] Setting key 0ff7a317b1bb2732e79f052afec3522b
I20260809 06:51:15.377426 28821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 06:51:15 dist-test-slave-jpj1 krb5kdc[28688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 06:51:15.391012 28821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1: Bootstrap starting.
I20260809 06:51:15.394114 28821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:15.395115 28821 log.cc:824] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:15.397889 28821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1: No bootstrap required, opened a new log
I20260809 06:51:15.398543 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:15.377841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:51789 (local address 127.0.0.1:40453)
0809 06:51:15.378623 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:15.378634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:15.378674 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:51:15.380369 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:15.380392 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:15.380789 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:15.381126 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:15.382892 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:15.384283 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:15.385952 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:15.386595 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:15.386733 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:15.391408 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:15.391415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:15.391430 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:15.391474 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:15.394431 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:15.395198 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:15.395211 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:15.395217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:15.395402 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:15.395882 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:15.395890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:15.395892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:15.396401 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:15.396732 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:15.397210 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 06:51:15.397509 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":333,"threads_started":1}
I20260809 06:51:15.402835 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:15.407886 28821 raft_consensus.cc:348] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11f1cae693b4479b87689e7b8c146a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260809 06:51:15.408259 28821 raft_consensus.cc:374] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:15.408357 28821 raft_consensus.cc:729] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11f1cae693b4479b87689e7b8c146a1, State: Initialized, Role: FOLLOWER
I20260809 06:51:15.408972 28821 consensus_queue.cc:260] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11f1cae693b4479b87689e7b8c146a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260809 06:51:15.409286 28821 raft_consensus.cc:388] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:15.409471 28821 raft_consensus.cc:482] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:15.409663 28821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:15.412272 28821 raft_consensus.cc:504] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11f1cae693b4479b87689e7b8c146a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260809 06:51:15.412840 28821 leader_election.cc:302] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f11f1cae693b4479b87689e7b8c146a1; no voters:
I20260809 06:51:15.413409 28821 leader_election.cc:288] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:15.413537 28833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:15.414148 28833 raft_consensus.cc:686] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [term 1 LEADER]: Becoming Leader. State: Replica: f11f1cae693b4479b87689e7b8c146a1, State: Running, Role: LEADER
I20260809 06:51:15.414772 28833 consensus_queue.cc:237] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11f1cae693b4479b87689e7b8c146a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260809 06:51:15.417689 28821 sys_catalog.cc:564] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:15.418048 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f11f1cae693b4479b87689e7b8c146a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11f1cae693b4479b87689e7b8c146a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } } }
I20260809 06:51:15.418063 28834 sys_catalog.cc:455] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f11f1cae693b4479b87689e7b8c146a1. Latest consensus state: current_term: 1 leader_uuid: "f11f1cae693b4479b87689e7b8c146a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f11f1cae693b4479b87689e7b8c146a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } } }
I20260809 06:51:15.418545 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:15.418545 28834 sys_catalog.cc:458] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:15.426841 28839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:15.427100 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:15.409189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:42095 (local address 127.0.0.1:40453)
0809 06:51:15.409347 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:15.409352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:15.409861 (+ 509us) server_negotiation.cc:407] Connection header received
0809 06:51:15.410019 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:15.410024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:15.410099 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:15.410200 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:15.411505 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:15.412603 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:15.414191 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:15.414908 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:15.415006 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:15.418418 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:15.418424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:15.418429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:15.418469 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:15.422453 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:15.423470 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:15.423476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:15.423480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:15.423701 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:15.424280 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:15.424285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:15.424289 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:15.424693 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:15.424979 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:15.426387 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0809 06:51:15.426601 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 06:51:15.433003 28839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:15.441820 28839 catalog_manager.cc:1347] Generated new cluster ID: 8bc254c416184f17942fcd1d5f08709f
I20260809 06:51:15.441918 28839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:15.455885 28839 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:15.457423 28839 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:15.466707 28839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f11f1cae693b4479b87689e7b8c146a1: Generated new TSK 0
I20260809 06:51:15.467749 28839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:15.480052 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28704
I20260809 06:51:15.481144 28812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:15.570259 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28704
W20260809 06:51:15.593816 28686 connection.cc:535] client connection to 127.0.0.1:40453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40453): Connection reset by peer (error 104)
2026-08-09T06:51:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2138 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:15 dist-test-slave-jpj1 krb5kdc[28863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:15 dist-test-slave-jpj1 krb5kdc[28863](info): set up 2 sockets
Aug 09 06:51:15 dist-test-slave-jpj1 krb5kdc[28863](info): commencing operation
krb5kdc: starting...
W20260809 06:51:16.780110 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:16 dist-test-slave-jpj1 krb5kdc[28863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:17.083276 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:17.424780 28884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:17.425109 28884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:17.425216 28884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:17.432606 28884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:17.432739 28884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:17.432787 28884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:17.432852 28884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:17.432896 28884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:17.446182 28884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:17.448524 28884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:17.450922 28884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:17.473232 28892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:17.473300 28890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:17.473304 28889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:17.476555 28884 server_base.cc:1034] running on GCE node
I20260809 06:51:17.478099 28884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:17.481223 28884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:17.482839 28884 hybrid_clock.cc:648] HybridClock initialized: now 1786258277482692 us; error 72 us; skew 500 ppm
I20260809 06:51:17.483608 28884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:17 dist-test-slave-jpj1 krb5kdc[28863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:17.489732 28884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:17.492990 28884 webserver.cc:460] Webserver started at http://127.0.0.1:46803/ using document root <none> and password file <none>
I20260809 06:51:17.494436 28884 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:17.494716 28884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:17.495296 28884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:17.501518 28884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "30fb3a024042482d8454b6bcae1e1213"
format_stamp: "Formatted at 2026-08-09 06:51:17 on dist-test-slave-jpj1"
server_key: "885bdd3c5dd43fa23f80a2a8cada985f"
server_key_iv: "d8fb522823ad7a838a25311097ae4e0f"
server_key_version: "encryptionkey@0"
I20260809 06:51:17.503137 28884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "30fb3a024042482d8454b6bcae1e1213"
format_stamp: "Formatted at 2026-08-09 06:51:17 on dist-test-slave-jpj1"
server_key: "885bdd3c5dd43fa23f80a2a8cada985f"
server_key_iv: "d8fb522823ad7a838a25311097ae4e0f"
server_key_version: "encryptionkey@0"
I20260809 06:51:17.513906 28884 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260809 06:51:17.521232 28901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:17.524009 28884 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260809 06:51:17.524341 28884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "30fb3a024042482d8454b6bcae1e1213"
format_stamp: "Formatted at 2026-08-09 06:51:17 on dist-test-slave-jpj1"
server_key: "885bdd3c5dd43fa23f80a2a8cada985f"
server_key_iv: "d8fb522823ad7a838a25311097ae4e0f"
server_key_version: "encryptionkey@0"
I20260809 06:51:17.524736 28884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:17.543339 28884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:17.547477 28884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:17.547782 28884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:17.565909 28884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45629
I20260809 06:51:17.565935 28984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45629 every 8 connection(s)
I20260809 06:51:17.567276 28884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:17.571003 28986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:17.574242 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 28884
I20260809 06:51:17.574442 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:17.574982 28487 external_mini_cluster.cc:1428] Setting key a271f71677fe158815aa8882e0f0b275
I20260809 06:51:17.583040 28986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213: Bootstrap starting.
I20260809 06:51:17.586612 28986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:17.587781 28986 log.cc:824] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:17.588271 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:17.576814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:37379 (local address 127.0.0.1:45629)
0809 06:51:17.577537 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:17.577548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:17.577589 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:51:17.579230 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:17.579254 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:17.579721 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:17.580045 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:17.580610 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:17.581914 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:17.583779 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:17.584446 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:17.584591 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:17.585957 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:17.585964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:17.585978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:17.586504 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:17.586574 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:17.586602 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:17.586732 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:17.587021 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 06:51:17.587212 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":339,"threads_started":1}
I20260809 06:51:17.590965 28986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213: No bootstrap required, opened a new log
I20260809 06:51:17.592378 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:17.598937 28986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fb3a024042482d8454b6bcae1e1213" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260809 06:51:17.599290 28986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:17.599391 28986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fb3a024042482d8454b6bcae1e1213, State: Initialized, Role: FOLLOWER
I20260809 06:51:17.599999 28986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fb3a024042482d8454b6bcae1e1213" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260809 06:51:17.600232 28986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:17.600373 28986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:17.600534 28986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:17.602557 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:17.596461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:51365 (local address 127.0.0.1:45629)
0809 06:51:17.596623 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:17.596627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:17.597071 (+ 444us) server_negotiation.cc:407] Connection header received
0809 06:51:17.597253 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:17.597260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:17.597322 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:17.597410 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:17.597913 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:17.598675 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:17.600055 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:17.600605 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:17.600723 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:17.601794 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:17.601799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:17.601802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:17.601818 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:17.601874 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:17.601889 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:17.601983 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:17.602239 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 06:51:17.602290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 06:51:17.603217 28986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fb3a024042482d8454b6bcae1e1213" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260809 06:51:17.603822 28986 leader_election.cc:302] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30fb3a024042482d8454b6bcae1e1213; no voters:
I20260809 06:51:17.604440 28986 leader_election.cc:288] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:17.604616 29002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:17.606530 29002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [term 1 LEADER]: Becoming Leader. State: Replica: 30fb3a024042482d8454b6bcae1e1213, State: Running, Role: LEADER
I20260809 06:51:17.606606 28986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:17.607021 29002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fb3a024042482d8454b6bcae1e1213" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } }
I20260809 06:51:17.610474 29003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30fb3a024042482d8454b6bcae1e1213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fb3a024042482d8454b6bcae1e1213" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } } }
I20260809 06:51:17.610441 29004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30fb3a024042482d8454b6bcae1e1213. Latest consensus state: current_term: 1 leader_uuid: "30fb3a024042482d8454b6bcae1e1213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fb3a024042482d8454b6bcae1e1213" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45629 } } }
I20260809 06:51:17.610921 29004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:17.610922 29003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:17.611950 29014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:17.618228 29014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:17.626788 29014 catalog_manager.cc:1347] Generated new cluster ID: c6e36b184d7a49f29461592698e87cc0
I20260809 06:51:17.626892 29014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:17.644639 29014 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:17.645864 29014 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:17.653777 29014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30fb3a024042482d8454b6bcae1e1213: Generated new TSK 0
I20260809 06:51:17.654531 29014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:17.669674 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28884
W20260809 06:51:17.670832 28976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:37379
W20260809 06:51:17.671491 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 28884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:17.671656 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 28884
W20260809 06:51:17.695158 28859 connection.cc:535] client connection to 127.0.0.1:45629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45629): Connection reset by peer (error 104)
2026-08-09T06:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2097 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:17 dist-test-slave-jpj1 krb5kdc[29030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:17 dist-test-slave-jpj1 krb5kdc[29030](info): set up 2 sockets
Aug 09 06:51:17 dist-test-slave-jpj1 krb5kdc[29030](info): commencing operation
krb5kdc: starting...
W20260809 06:51:18.889025 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:19 dist-test-slave-jpj1 krb5kdc[29030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 06:51:19.184219 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:19.525202 29052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:19.525511 29052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:19.525653 29052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:19.532735 29052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:19.532860 29052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:19.532943 29052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:19.533030 29052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:19.533093 29052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:19.542667 29052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:19.543959 29052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:19.545289 29052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:19.556427 29071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:19.556483 29068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:19.556522 29069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:19.557307 29052 server_base.cc:1034] running on GCE node
I20260809 06:51:19.558043 29052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:19.559356 29052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:19.560549 29052 hybrid_clock.cc:648] HybridClock initialized: now 1786258279560499 us; error 39 us; skew 500 ppm
I20260809 06:51:19.560873 29052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:19 dist-test-slave-jpj1 krb5kdc[29030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:19.567852 29052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 06:51:19.569195 29052 webserver.cc:460] Webserver started at http://127.0.0.1:37393/ using document root <none> and password file <none>
I20260809 06:51:19.569867 29052 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:19.569984 29052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:19.570269 29052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:19.573160 29052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "b5042c73564f40e284424ef45ba01964"
format_stamp: "Formatted at 2026-08-09 06:51:19 on dist-test-slave-jpj1"
server_key: "267dfa62f7b0c69ab2feaff759eec227"
server_key_iv: "7ff808cc7d2f2c2ecb4affb36766dd96"
server_key_version: "encryptionkey@0"
I20260809 06:51:19.573875 29052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "b5042c73564f40e284424ef45ba01964"
format_stamp: "Formatted at 2026-08-09 06:51:19 on dist-test-slave-jpj1"
server_key: "267dfa62f7b0c69ab2feaff759eec227"
server_key_iv: "7ff808cc7d2f2c2ecb4affb36766dd96"
server_key_version: "encryptionkey@0"
I20260809 06:51:19.578685 29052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260809 06:51:19.581959 29085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:19.583129 29052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 06:51:19.583273 29052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "b5042c73564f40e284424ef45ba01964"
format_stamp: "Formatted at 2026-08-09 06:51:19 on dist-test-slave-jpj1"
server_key: "267dfa62f7b0c69ab2feaff759eec227"
server_key_iv: "7ff808cc7d2f2c2ecb4affb36766dd96"
server_key_version: "encryptionkey@0"
I20260809 06:51:19.583449 29052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:19.588783 29052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:19.592419 29052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:19.592672 29052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:19.609558 29052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35267
I20260809 06:51:19.609616 29167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35267 every 8 connection(s)
I20260809 06:51:19.610868 29052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:19.612250 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29052
I20260809 06:51:19.612449 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:19.613003 28487 external_mini_cluster.cc:1428] Setting key 0c57d048dd9aecb098d485dd73c4e80d
I20260809 06:51:19.615317 29170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:19.620244 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:19.614696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:41493 (local address 127.0.0.1:35267)
0809 06:51:19.615459 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:19.615470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:19.615512 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:51:19.617076 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:19.617097 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:19.617415 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:19.617726 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:19.618716 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:19.618722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:19.618733 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:19.618753 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:19.618828 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:19.618848 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:19.619013 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:19.619136 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 06:51:19.619326 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260809 06:51:19.623557 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:19.627485 29170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964: Bootstrap starting.
I20260809 06:51:19.630949 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:19.628710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.88:35645 (local address 127.0.0.1:35267)
0809 06:51:19.628835 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:19.628839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:19.629231 (+ 392us) server_negotiation.cc:407] Connection header received
0809 06:51:19.629403 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:19.629407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:19.629453 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:19.629522 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:19.630401 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:19.630405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:19.630408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:19.630421 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:19.630455 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:19.630468 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:19.630551 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:19.630672 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 06:51:19.630711 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 06:51:19.631276 29170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:19.632364 29170 log.cc:824] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:19.634809 29170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964: No bootstrap required, opened a new log
I20260809 06:51:19.641702 29170 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5042c73564f40e284424ef45ba01964" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260809 06:51:19.642063 29170 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:19.642151 29170 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5042c73564f40e284424ef45ba01964, State: Initialized, Role: FOLLOWER
I20260809 06:51:19.642659 29170 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5042c73564f40e284424ef45ba01964" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260809 06:51:19.642890 29170 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:19.643026 29170 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:19.643182 29170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:19.645217 29170 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5042c73564f40e284424ef45ba01964" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260809 06:51:19.645658 29170 leader_election.cc:302] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5042c73564f40e284424ef45ba01964; no voters:
I20260809 06:51:19.646179 29170 leader_election.cc:288] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:19.646277 29188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:19.647379 29188 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [term 1 LEADER]: Becoming Leader. State: Replica: b5042c73564f40e284424ef45ba01964, State: Running, Role: LEADER
I20260809 06:51:19.647891 29188 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5042c73564f40e284424ef45ba01964" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } }
I20260809 06:51:19.648135 29170 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:19.650581 29189 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5042c73564f40e284424ef45ba01964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5042c73564f40e284424ef45ba01964" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } } }
I20260809 06:51:19.650553 29190 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5042c73564f40e284424ef45ba01964. Latest consensus state: current_term: 1 leader_uuid: "b5042c73564f40e284424ef45ba01964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5042c73564f40e284424ef45ba01964" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35267 } } }
I20260809 06:51:19.651136 29190 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:19.651137 29189 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:19.652098 29201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:19.657286 29201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:19.665720 29201 catalog_manager.cc:1347] Generated new cluster ID: c78c98e4b9154ec886b9e7d074de575f
I20260809 06:51:19.665822 29201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:19.674575 29201 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:19.675016 29201 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:19.682771 29201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5042c73564f40e284424ef45ba01964: Generated new TSK 0
I20260809 06:51:19.683557 29201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:19.691017 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29052
W20260809 06:51:19.692327 29162 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.88:41493
W20260809 06:51:19.693046 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29052: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:19.693215 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29052
2026-08-09T06:51:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:19 dist-test-slave-jpj1 krb5kdc[29221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:19 dist-test-slave-jpj1 krb5kdc[29221](info): set up 2 sockets
Aug 09 06:51:19 dist-test-slave-jpj1 krb5kdc[29221](info): commencing operation
krb5kdc: starting...
W20260809 06:51:20.908223 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:21 dist-test-slave-jpj1 krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:21.198953 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41093
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:42159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:21.538491 29243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:21.538775 29243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:21.538873 29243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:21.545787 29243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:21.545888 29243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:21.545940 29243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:21.546000 29243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:21.546043 29243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:21.555420 29243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:42159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:41093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:21.556614 29243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:21.557875 29243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:21.569285 29254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:21.569211 29255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:21.569244 29258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:21.570027 29243 server_base.cc:1034] running on GCE node
I20260809 06:51:21.570787 29243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:21.572180 29243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:21.573380 29243 hybrid_clock.cc:648] HybridClock initialized: now 1786258281573332 us; error 44 us; skew 500 ppm
I20260809 06:51:21.573715 29243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:21 dist-test-slave-jpj1 krb5kdc[29221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258281, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:21.576939 29243 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:21.578430 29243 webserver.cc:460] Webserver started at http://127.27.209.254:43821/ using document root <none> and password file <none>
I20260809 06:51:21.579109 29243 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:21.579236 29243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:21.579523 29243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:21.582432 29243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "19da71ec74054257ba63fe04880ba687"
format_stamp: "Formatted at 2026-08-09 06:51:21 on dist-test-slave-jpj1"
server_key: "ae2cbf5eb9f1b0cd6f29582d00fc87fe"
server_key_iv: "e888283ba547895f5b57ade55a261612"
server_key_version: "encryptionkey@0"
I20260809 06:51:21.583160 29243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "19da71ec74054257ba63fe04880ba687"
format_stamp: "Formatted at 2026-08-09 06:51:21 on dist-test-slave-jpj1"
server_key: "ae2cbf5eb9f1b0cd6f29582d00fc87fe"
server_key_iv: "e888283ba547895f5b57ade55a261612"
server_key_version: "encryptionkey@0"
I20260809 06:51:21.588038 29243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 06:51:21.591436 29267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:21.592713 29243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:51:21.592868 29243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "19da71ec74054257ba63fe04880ba687"
format_stamp: "Formatted at 2026-08-09 06:51:21 on dist-test-slave-jpj1"
server_key: "ae2cbf5eb9f1b0cd6f29582d00fc87fe"
server_key_iv: "e888283ba547895f5b57ade55a261612"
server_key_version: "encryptionkey@0"
I20260809 06:51:21.593039 29243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:21.609431 29243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:21.613540 29243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:21.613811 29243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:21.631088 29243 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:41093
I20260809 06:51:21.631126 29355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:41093 every 8 connection(s)
I20260809 06:51:21.632437 29243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:21.635485 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29243
I20260809 06:51:21.635700 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:21.636276 28487 external_mini_cluster.cc:1428] Setting key 8406957493db9ae7450372072ad6add4
I20260809 06:51:21.636430 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:21.648598 29356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687: Bootstrap starting.
Aug 09 06:51:21 dist-test-slave-jpj1 krb5kdc[29221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:21.652174 29356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:21.653257 29356 log.cc:824] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:21.656468 29356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687: No bootstrap required, opened a new log
I20260809 06:51:21.658953 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:21.638011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40803 (local address 127.27.209.254:41093)
0809 06:51:21.638779 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:21.638790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:21.638830 (+ 40us) server_negotiation.cc:407] Connection header received
0809 06:51:21.640989 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:21.641012 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:21.641412 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:21.641798 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:21.643264 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:21.644565 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:21.645854 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:21.646550 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:21.646750 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:21.651249 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:21.651258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:21.651271 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:21.651307 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:21.654686 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:21.655662 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:21.655668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:21.655673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:21.655853 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:21.656330 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:21.656333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:21.656335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:21.656876 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:21.657126 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:21.657615 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 06:51:21.657917 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":356,"threads_started":1}
I20260809 06:51:21.662735 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:21.665247 29356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da71ec74054257ba63fe04880ba687" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41093 } }
I20260809 06:51:21.665694 29356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:21.665814 29356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19da71ec74054257ba63fe04880ba687, State: Initialized, Role: FOLLOWER
I20260809 06:51:21.666499 29356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da71ec74054257ba63fe04880ba687" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41093 } }
I20260809 06:51:21.666811 29356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:21.666951 29356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:21.667116 29356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:21.670312 29356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da71ec74054257ba63fe04880ba687" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41093 } }
I20260809 06:51:21.670957 29356 leader_election.cc:302] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19da71ec74054257ba63fe04880ba687; no voters:
I20260809 06:51:21.671555 29356 leader_election.cc:288] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:21.671698 29369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:21.672812 29369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [term 1 LEADER]: Becoming Leader. State: Replica: 19da71ec74054257ba63fe04880ba687, State: Running, Role: LEADER
I20260809 06:51:21.673408 29369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da71ec74054257ba63fe04880ba687" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41093 } }
I20260809 06:51:21.673735 29356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:21.676191 29370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19da71ec74054257ba63fe04880ba687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da71ec74054257ba63fe04880ba687" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41093 } } }
I20260809 06:51:21.676569 29370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:21.678778 29371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19da71ec74054257ba63fe04880ba687. Latest consensus state: current_term: 1 leader_uuid: "19da71ec74054257ba63fe04880ba687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19da71ec74054257ba63fe04880ba687" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41093 } } }
I20260809 06:51:21.679144 29371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:21.680636 29375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:21.685910 29375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:21.686144 29360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:21.668064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44329 (local address 127.27.209.254:41093)
0809 06:51:21.668293 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:21.668300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:21.668668 (+ 368us) server_negotiation.cc:407] Connection header received
0809 06:51:21.668823 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:21.668827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:21.668890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:21.668975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:21.670871 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:21.671874 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:21.673664 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:21.674279 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:21.674409 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:21.677697 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:21.677703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:21.677709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:21.677749 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:21.681512 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:21.682520 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:21.682524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:21.682532 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:21.682687 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:21.683305 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:21.683311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:21.683314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:21.683806 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:21.685273 (+ 1467us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:21.685467 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 06:51:21.685663 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 06:51:21.696082 29375 catalog_manager.cc:1347] Generated new cluster ID: 65dfe0ab3ad04c9e8795c9aea6b94219
I20260809 06:51:21.696175 29375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:21.712906 29375 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:21.714354 29375 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:21.720321 29375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19da71ec74054257ba63fe04880ba687: Generated new TSK 0
I20260809 06:51:21.721315 29375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:21.733485 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29243
I20260809 06:51:21.734949 29347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:21.824409 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29243
W20260809 06:51:21.846144 29218 connection.cc:535] client connection to 127.27.209.254:41093 recv error: Network error: failed to read from TLS socket (remote: 127.27.209.254:41093): Connection reset by peer (error 104)
2026-08-09T06:51:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2128 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:21 dist-test-slave-jpj1 krb5kdc[29398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:21 dist-test-slave-jpj1 krb5kdc[29398](info): set up 2 sockets
Aug 09 06:51:21 dist-test-slave-jpj1 krb5kdc[29398](info): commencing operation
krb5kdc: starting...
W20260809 06:51:23.035573 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:23 dist-test-slave-jpj1 krb5kdc[29398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:23.331008 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41231
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:23.669641 29425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:23.669924 29425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:23.670017 29425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:23.677048 29425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:23.677162 29425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:23.677217 29425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:23.677296 29425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:23.677342 29425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:23.686985 29425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:41231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:23.688244 29425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:23.689640 29425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:23.700913 29434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:23.701004 29431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:23.700976 29432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:23.701849 29425 server_base.cc:1034] running on GCE node
I20260809 06:51:23.702623 29425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:23.704070 29425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:23.705266 29425 hybrid_clock.cc:648] HybridClock initialized: now 1786258283705213 us; error 43 us; skew 500 ppm
I20260809 06:51:23.705603 29425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:23 dist-test-slave-jpj1 krb5kdc[29398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:23.708778 29425 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:23.710157 29425 webserver.cc:460] Webserver started at http://127.27.209.254:33605/ using document root <none> and password file <none>
I20260809 06:51:23.710815 29425 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:23.710939 29425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:23.711225 29425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:23.714294 29425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "969e21c12667465faa891613336b9d47"
format_stamp: "Formatted at 2026-08-09 06:51:23 on dist-test-slave-jpj1"
server_key: "78bfaa24372237f3e62beed9b8d41bb1"
server_key_iv: "5ca7d5863d61ac390f0bf746b5e0ee4a"
server_key_version: "encryptionkey@0"
I20260809 06:51:23.715078 29425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "969e21c12667465faa891613336b9d47"
format_stamp: "Formatted at 2026-08-09 06:51:23 on dist-test-slave-jpj1"
server_key: "78bfaa24372237f3e62beed9b8d41bb1"
server_key_iv: "5ca7d5863d61ac390f0bf746b5e0ee4a"
server_key_version: "encryptionkey@0"
I20260809 06:51:23.720057 29425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 06:51:23.723345 29445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:23.724670 29425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:23.724825 29425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "969e21c12667465faa891613336b9d47"
format_stamp: "Formatted at 2026-08-09 06:51:23 on dist-test-slave-jpj1"
server_key: "78bfaa24372237f3e62beed9b8d41bb1"
server_key_iv: "5ca7d5863d61ac390f0bf746b5e0ee4a"
server_key_version: "encryptionkey@0"
I20260809 06:51:23.725001 29425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:23.751106 29425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:23.755508 29425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:23.755810 29425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:23.773483 29425 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:41231
I20260809 06:51:23.773502 29528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:41231 every 8 connection(s)
I20260809 06:51:23.774847 29425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:23.778682 29529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:23.781025 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29425
I20260809 06:51:23.781205 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:23.781733 28487 external_mini_cluster.cc:1428] Setting key 5295800e1d081dd9cc01c4f392fe319b
I20260809 06:51:23.790657 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47: Bootstrap starting.
I20260809 06:51:23.794066 29529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:23.794921 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:23.783584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35761 (local address 127.27.209.254:41231)
0809 06:51:23.784358 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:23.784369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:23.784414 (+ 45us) server_negotiation.cc:407] Connection header received
0809 06:51:23.786034 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:23.786057 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:23.786429 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:23.786735 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:23.787302 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:23.788738 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:23.790503 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:23.791191 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:23.791328 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:23.792586 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:23.792594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:23.792612 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:23.793120 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:23.793196 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:23.793220 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:23.793340 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:23.793594 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 06:51:23.793900 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":312,"threads_started":1}
I20260809 06:51:23.795130 29529 log.cc:824] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:23.798244 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47: No bootstrap required, opened a new log
I20260809 06:51:23.798904 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:23.807901 29529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969e21c12667465faa891613336b9d47" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41231 } }
I20260809 06:51:23.808372 29529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:23.808483 29529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969e21c12667465faa891613336b9d47, State: Initialized, Role: FOLLOWER
I20260809 06:51:23.809099 29529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969e21c12667465faa891613336b9d47" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41231 } }
I20260809 06:51:23.809325 29529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:23.809458 29529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:23.809662 29529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:23.810622 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:23.804088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53691 (local address 127.27.209.254:41231)
0809 06:51:23.804262 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:23.804268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:23.804643 (+ 375us) server_negotiation.cc:407] Connection header received
0809 06:51:23.804786 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:23.804790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:23.804866 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:23.804961 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:23.805445 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:23.806596 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:23.807982 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:23.808623 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:23.808742 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:23.809851 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:23.809857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:23.809862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:23.809879 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:23.809940 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:23.809959 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:23.810043 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:23.810248 (+ 205us) server_negotiation.cc:299] Negotiation successful
0809 06:51:23.810293 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:51:23.812417 29529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969e21c12667465faa891613336b9d47" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41231 } }
I20260809 06:51:23.812881 29529 leader_election.cc:302] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 969e21c12667465faa891613336b9d47; no voters:
I20260809 06:51:23.813431 29529 leader_election.cc:288] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:23.813517 29547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:23.814311 29547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [term 1 LEADER]: Becoming Leader. State: Replica: 969e21c12667465faa891613336b9d47, State: Running, Role: LEADER
I20260809 06:51:23.814888 29547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969e21c12667465faa891613336b9d47" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41231 } }
I20260809 06:51:23.816723 29547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 969e21c12667465faa891613336b9d47. Latest consensus state: current_term: 1 leader_uuid: "969e21c12667465faa891613336b9d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969e21c12667465faa891613336b9d47" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41231 } } }
I20260809 06:51:23.817026 29547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:23.817514 29550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "969e21c12667465faa891613336b9d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "969e21c12667465faa891613336b9d47" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41231 } } }
I20260809 06:51:23.817899 29550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:23.818390 29529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:23.819741 29553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:23.825160 29553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:23.835153 29553 catalog_manager.cc:1347] Generated new cluster ID: ebd62e50614345459c9e1fe513107fbc
I20260809 06:51:23.835340 29553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:23.864734 29553 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:23.866027 29553 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:23.873761 29553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 969e21c12667465faa891613336b9d47: Generated new TSK 0
I20260809 06:51:23.874570 29553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:23.883759 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29425
W20260809 06:51:23.884866 29524 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35761
W20260809 06:51:23.885573 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:23.885710 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29425
W20260809 06:51:23.913057 29396 connection.cc:535] client connection to 127.27.209.254:41231 recv error: Network error: failed to read from TLS socket (remote: 127.27.209.254:41231): Connection reset by peer (error 104)
2026-08-09T06:51:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:23 dist-test-slave-jpj1 krb5kdc[29580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:23 dist-test-slave-jpj1 krb5kdc[29580](info): set up 2 sockets
Aug 09 06:51:23 dist-test-slave-jpj1 krb5kdc[29580](info): commencing operation
krb5kdc: starting...
W20260809 06:51:25.114853 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:25 dist-test-slave-jpj1 krb5kdc[29580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:25.408452 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40743
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:40743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:25.746857 29603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:25.747143 29603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:25.747272 29603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:25.754438 29603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:25.754554 29603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:25.754642 29603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:25.754719 29603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:25.754777 29603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:25.764566 29603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:40743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:40743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:25.765786 29603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:25.767127 29603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:25.778395 29613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:25.778384 29611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:25.778447 29610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:25.779245 29603 server_base.cc:1034] running on GCE node
I20260809 06:51:25.780104 29603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:25.781441 29603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:25.782644 29603 hybrid_clock.cc:648] HybridClock initialized: now 1786258285782603 us; error 32 us; skew 500 ppm
I20260809 06:51:25.782989 29603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:25 dist-test-slave-jpj1 krb5kdc[29580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:25.789830 29603 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:25.791220 29603 webserver.cc:460] Webserver started at http://127.27.209.254:45309/ using document root <none> and password file <none>
I20260809 06:51:25.791944 29603 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:25.792070 29603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:25.792366 29603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:25.795279 29603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "0a38c858681f4f3db32d19e266fd6226"
format_stamp: "Formatted at 2026-08-09 06:51:25 on dist-test-slave-jpj1"
server_key: "48db28e4295c2c235ce0b9aa0c6df71d"
server_key_iv: "e9e1f204c202fb56a4dc1d7a536cd007"
server_key_version: "encryptionkey@0"
I20260809 06:51:25.796079 29603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "0a38c858681f4f3db32d19e266fd6226"
format_stamp: "Formatted at 2026-08-09 06:51:25 on dist-test-slave-jpj1"
server_key: "48db28e4295c2c235ce0b9aa0c6df71d"
server_key_iv: "e9e1f204c202fb56a4dc1d7a536cd007"
server_key_version: "encryptionkey@0"
I20260809 06:51:25.801443 29603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:51:25.804863 29623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:25.806217 29603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:51:25.806383 29603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "0a38c858681f4f3db32d19e266fd6226"
format_stamp: "Formatted at 2026-08-09 06:51:25 on dist-test-slave-jpj1"
server_key: "48db28e4295c2c235ce0b9aa0c6df71d"
server_key_iv: "e9e1f204c202fb56a4dc1d7a536cd007"
server_key_version: "encryptionkey@0"
I20260809 06:51:25.806565 29603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:25.812318 29603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:25.816218 29603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:25.816516 29603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:25.834080 29603 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:40743
I20260809 06:51:25.834112 29704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:40743 every 8 connection(s)
I20260809 06:51:25.835460 29603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:25.838549 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29603
I20260809 06:51:25.838730 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:25.839286 28487 external_mini_cluster.cc:1428] Setting key 62f102ce0376060976ca93802647dd37
I20260809 06:51:25.839581 29705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:25.847290 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:25.841109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45367 (local address 127.27.209.254:40743)
0809 06:51:25.841895 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:25.841906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:25.841947 (+ 41us) server_negotiation.cc:407] Connection header received
0809 06:51:25.844130 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:25.844152 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:25.844483 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:25.844814 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:25.845824 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:25.845830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:25.845846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:25.845860 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:25.845919 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:25.845939 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:25.846110 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:25.846203 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 06:51:25.846376 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":361,"threads_started":1}
I20260809 06:51:25.851068 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:25.853364 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226: Bootstrap starting.
I20260809 06:51:25.857169 29705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:25.858410 29710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:25.856160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57297 (local address 127.27.209.254:40743)
0809 06:51:25.856348 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:25.856355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:25.856727 (+ 372us) server_negotiation.cc:407] Connection header received
0809 06:51:25.856848 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:25.856854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:25.856913 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:25.857020 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:25.857825 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:25.857829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:25.857832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:25.857847 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:25.857884 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:25.857899 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:25.857981 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:25.858109 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 06:51:25.858153 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:51:25.858440 29705 log.cc:824] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:25.861197 29705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226: No bootstrap required, opened a new log
I20260809 06:51:25.868453 29705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a38c858681f4f3db32d19e266fd6226" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40743 } }
I20260809 06:51:25.868794 29705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:25.868878 29705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a38c858681f4f3db32d19e266fd6226, State: Initialized, Role: FOLLOWER
I20260809 06:51:25.869405 29705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a38c858681f4f3db32d19e266fd6226" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40743 } }
I20260809 06:51:25.869628 29705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:25.869728 29705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:25.869859 29705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:25.871938 29705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a38c858681f4f3db32d19e266fd6226" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40743 } }
I20260809 06:51:25.872377 29705 leader_election.cc:302] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a38c858681f4f3db32d19e266fd6226; no voters:
I20260809 06:51:25.872916 29705 leader_election.cc:288] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:25.873041 29724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:25.873648 29724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [term 1 LEADER]: Becoming Leader. State: Replica: 0a38c858681f4f3db32d19e266fd6226, State: Running, Role: LEADER
I20260809 06:51:25.874145 29724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a38c858681f4f3db32d19e266fd6226" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40743 } }
I20260809 06:51:25.874828 29705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:25.876893 29726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a38c858681f4f3db32d19e266fd6226. Latest consensus state: current_term: 1 leader_uuid: "0a38c858681f4f3db32d19e266fd6226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a38c858681f4f3db32d19e266fd6226" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40743 } } }
I20260809 06:51:25.877305 29726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:25.876865 29725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a38c858681f4f3db32d19e266fd6226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a38c858681f4f3db32d19e266fd6226" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 40743 } } }
I20260809 06:51:25.877907 29725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:25.878937 29731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:25.883949 29731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:25.893191 29731 catalog_manager.cc:1347] Generated new cluster ID: ef79a3ce69b2470397f24719a0b57497
I20260809 06:51:25.893299 29731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:25.904397 29731 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:25.904798 29731 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:25.915877 29731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a38c858681f4f3db32d19e266fd6226: Generated new TSK 0
I20260809 06:51:25.916627 29731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:25.921617 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29603
W20260809 06:51:25.922789 29700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45367
W20260809 06:51:25.923445 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29603: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:25.923600 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29603
2026-08-09T06:51:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:26 dist-test-slave-jpj1 krb5kdc[29755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:26 dist-test-slave-jpj1 krb5kdc[29755](info): set up 2 sockets
Aug 09 06:51:26 dist-test-slave-jpj1 krb5kdc[29755](info): commencing operation
krb5kdc: starting...
W20260809 06:51:27.140779 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.003s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:27 dist-test-slave-jpj1 krb5kdc[29755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:27.437184 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:39309
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:40177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:39309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:27.776348 29774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:27.776672 29774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:27.776813 29774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:27.783902 29774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:27.784027 29774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:27.784108 29774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:27.784204 29774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:27.784265 29774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:27.793989 29774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:40177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:39309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:27.795315 29774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:27.796674 29774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:27.808050 29779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:27.808115 29780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:27.808236 29782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:27.809183 29774 server_base.cc:1034] running on GCE node
I20260809 06:51:27.809973 29774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:27.811390 29774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:27.812599 29774 hybrid_clock.cc:648] HybridClock initialized: now 1786258287812537 us; error 50 us; skew 500 ppm
I20260809 06:51:27.812973 29774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:27 dist-test-slave-jpj1 krb5kdc[29755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:27.816259 29774 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:27.817677 29774 webserver.cc:460] Webserver started at http://127.27.209.254:43685/ using document root <none> and password file <none>
I20260809 06:51:27.818356 29774 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:27.818466 29774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:27.818761 29774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:27.821753 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "01824fb329ee44ce972545a434ef145d"
format_stamp: "Formatted at 2026-08-09 06:51:27 on dist-test-slave-jpj1"
server_key: "d48f3a635367a931161d2c8226d64762"
server_key_iv: "a3610ce2ba61f46ba867b9422221f9bf"
server_key_version: "encryptionkey@0"
I20260809 06:51:27.822496 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "01824fb329ee44ce972545a434ef145d"
format_stamp: "Formatted at 2026-08-09 06:51:27 on dist-test-slave-jpj1"
server_key: "d48f3a635367a931161d2c8226d64762"
server_key_iv: "a3610ce2ba61f46ba867b9422221f9bf"
server_key_version: "encryptionkey@0"
I20260809 06:51:27.827605 29774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 06:51:27.830976 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:27.832288 29774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 06:51:27.832445 29774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "01824fb329ee44ce972545a434ef145d"
format_stamp: "Formatted at 2026-08-09 06:51:27 on dist-test-slave-jpj1"
server_key: "d48f3a635367a931161d2c8226d64762"
server_key_iv: "a3610ce2ba61f46ba867b9422221f9bf"
server_key_version: "encryptionkey@0"
I20260809 06:51:27.832616 29774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:27.848248 29774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:27.852097 29774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:27.852342 29774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:27.869876 29774 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:39309
I20260809 06:51:27.869915 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:39309 every 8 connection(s)
I20260809 06:51:27.871166 29774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:27.874862 29874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:27.876526 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29774
I20260809 06:51:27.876680 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:27.877106 28487 external_mini_cluster.cc:1428] Setting key fea51049794d831b3c3706a80cfc6d48
I20260809 06:51:27.888446 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d: Bootstrap starting.
Aug 09 06:51:27 dist-test-slave-jpj1 krb5kdc[29755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:27.892246 29874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:27.893312 29874 log.cc:824] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:27.896113 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d: No bootstrap required, opened a new log
I20260809 06:51:27.899405 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:27.878832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47911 (local address 127.27.209.254:39309)
0809 06:51:27.879679 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:27.879690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:27.879742 (+ 52us) server_negotiation.cc:407] Connection header received
0809 06:51:27.881366 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:27.881389 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:27.881796 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:27.882088 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:27.883891 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:27.885221 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:27.887051 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:27.887780 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:27.887904 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:27.892602 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:27.892613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:27.892627 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:27.892674 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:27.895483 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:27.896158 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:27.896165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:27.896171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:27.896352 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:27.896714 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:27.896720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:27.896723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:27.897235 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:27.897510 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:27.897984 (+ 474us) server_negotiation.cc:299] Negotiation successful
0809 06:51:27.898386 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":354,"threads_started":1}
I20260809 06:51:27.903093 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:27.904031 29874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01824fb329ee44ce972545a434ef145d" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39309 } }
I20260809 06:51:27.904371 29874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:27.904457 29874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01824fb329ee44ce972545a434ef145d, State: Initialized, Role: FOLLOWER
I20260809 06:51:27.904968 29874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01824fb329ee44ce972545a434ef145d" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39309 } }
I20260809 06:51:27.905200 29874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:27.905318 29874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:27.905475 29874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:27.908097 29874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01824fb329ee44ce972545a434ef145d" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39309 } }
I20260809 06:51:27.908703 29874 leader_election.cc:302] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01824fb329ee44ce972545a434ef145d; no voters:
I20260809 06:51:27.909364 29874 leader_election.cc:288] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:27.909456 29888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:27.911353 29874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:27.911916 29888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [term 1 LEADER]: Becoming Leader. State: Replica: 01824fb329ee44ce972545a434ef145d, State: Running, Role: LEADER
I20260809 06:51:27.912514 29888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01824fb329ee44ce972545a434ef145d" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39309 } }
I20260809 06:51:27.915093 29895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01824fb329ee44ce972545a434ef145d. Latest consensus state: current_term: 1 leader_uuid: "01824fb329ee44ce972545a434ef145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01824fb329ee44ce972545a434ef145d" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39309 } } }
I20260809 06:51:27.915469 29895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:27.915769 29889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01824fb329ee44ce972545a434ef145d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01824fb329ee44ce972545a434ef145d" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 39309 } } }
I20260809 06:51:27.916344 29889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:27.925011 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:27.907327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55271 (local address 127.27.209.254:39309)
0809 06:51:27.907468 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:27.907472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:27.908019 (+ 547us) server_negotiation.cc:407] Connection header received
0809 06:51:27.908176 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:27.908182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:27.908247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:27.908334 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:27.910382 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:27.911584 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:27.913287 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:27.914035 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:27.914180 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:27.917740 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:27.917748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:27.917753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:27.917794 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:27.921463 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:27.922345 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:27.922352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:27.922356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:27.922508 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:27.922978 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:27.922984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:27.922988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:27.923487 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:27.923775 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:27.924417 (+ 642us) server_negotiation.cc:299] Negotiation successful
0809 06:51:27.924648 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 06:51:27.928440 29901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:27.933607 29901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:27.943096 29901 catalog_manager.cc:1347] Generated new cluster ID: 09da1bb8176141e294676fb79430f801
I20260809 06:51:27.943238 29901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:27.964377 29901 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:27.966285 29901 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:27.975589 29901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01824fb329ee44ce972545a434ef145d: Generated new TSK 0
I20260809 06:51:27.976591 29901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:27.991899 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29774
I20260809 06:51:27.993448 29864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:28.090282 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29774
2026-08-09T06:51:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2161 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:28 dist-test-slave-jpj1 krb5kdc[29928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:28 dist-test-slave-jpj1 krb5kdc[29928](info): set up 2 sockets
Aug 09 06:51:28 dist-test-slave-jpj1 krb5kdc[29928](info): commencing operation
krb5kdc: starting...
W20260809 06:51:29.324842 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:29 dist-test-slave-jpj1 krb5kdc[29928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:29.619503 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41443
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:29.959861 29952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:29.960158 29952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:29.960292 29952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:29.967274 29952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:29.967386 29952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:29.967466 29952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:29.967574 29952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:29.967638 29952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:29.977062 29952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:29.978283 29952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:29.979580 29952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:29.990710 29962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:29.990758 29958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:29.990787 29959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:29.991725 29952 server_base.cc:1034] running on GCE node
I20260809 06:51:29.992486 29952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:29.993764 29952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:29.994956 29952 hybrid_clock.cc:648] HybridClock initialized: now 1786258289994917 us; error 34 us; skew 500 ppm
I20260809 06:51:29.995280 29952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:29 dist-test-slave-jpj1 krb5kdc[29928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:29.998194 29952 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:29.999629 29952 webserver.cc:460] Webserver started at http://127.27.209.254:41249/ using document root <none> and password file <none>
I20260809 06:51:30.000288 29952 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:30.000391 29952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:30.000633 29952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:30.003520 29952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "8f6a797e8a534274b29ef79651d40e53"
format_stamp: "Formatted at 2026-08-09 06:51:29 on dist-test-slave-jpj1"
server_key: "41dd8778a8ce2500e337d08cc00cd5da"
server_key_iv: "dfcd9201d79cf8460cabeb8352957543"
server_key_version: "encryptionkey@0"
I20260809 06:51:30.004311 29952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "8f6a797e8a534274b29ef79651d40e53"
format_stamp: "Formatted at 2026-08-09 06:51:29 on dist-test-slave-jpj1"
server_key: "41dd8778a8ce2500e337d08cc00cd5da"
server_key_iv: "dfcd9201d79cf8460cabeb8352957543"
server_key_version: "encryptionkey@0"
I20260809 06:51:30.009379 29952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 06:51:30.012708 29973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:30.013973 29952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:51:30.014119 29952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "8f6a797e8a534274b29ef79651d40e53"
format_stamp: "Formatted at 2026-08-09 06:51:29 on dist-test-slave-jpj1"
server_key: "41dd8778a8ce2500e337d08cc00cd5da"
server_key_iv: "dfcd9201d79cf8460cabeb8352957543"
server_key_version: "encryptionkey@0"
I20260809 06:51:30.014289 29952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:30.031751 29952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:30.035497 29952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:30.035764 29952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:30.052915 29952 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:41443
I20260809 06:51:30.052939 30047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:41443 every 8 connection(s)
I20260809 06:51:30.054219 29952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:30.057909 30051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:30.058182 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 29952
I20260809 06:51:30.058374 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:30.058941 28487 external_mini_cluster.cc:1428] Setting key 6bf7ad5282e40f2ac91dfaa6ea26fff0
I20260809 06:51:30.071214 30051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53: Bootstrap starting.
I20260809 06:51:30.072865 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:30.060715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55543 (local address 127.27.209.254:41443)
0809 06:51:30.061511 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:30.061523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:30.061565 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:51:30.063979 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:30.064003 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:30.064389 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:30.064718 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:30.065280 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:30.066488 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:30.068360 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:30.069148 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:30.069285 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:30.070532 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:30.070540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:30.070555 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:30.071097 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:30.071178 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:30.071207 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:30.071294 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:30.071607 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 06:51:30.071904 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":370,"threads_started":1}
I20260809 06:51:30.074466 30051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:30.075628 30051 log.cc:824] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:30.076543 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:30.078718 30051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53: No bootstrap required, opened a new log
I20260809 06:51:30.088049 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:30.081569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45485 (local address 127.27.209.254:41443)
0809 06:51:30.081761 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:30.081767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:30.082243 (+ 476us) server_negotiation.cc:407] Connection header received
0809 06:51:30.082393 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:30.082398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:30.082481 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:30.082583 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:30.083109 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:30.084248 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:30.085526 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:30.086221 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:30.086347 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:30.087311 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:30.087317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:30.087321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:30.087341 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:30.087400 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:30.087421 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:30.087511 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:30.087649 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 06:51:30.087748 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 06:51:30.088092 30051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6a797e8a534274b29ef79651d40e53" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41443 } }
I20260809 06:51:30.088500 30051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:30.088618 30051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6a797e8a534274b29ef79651d40e53, State: Initialized, Role: FOLLOWER
I20260809 06:51:30.089154 30051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6a797e8a534274b29ef79651d40e53" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41443 } }
I20260809 06:51:30.089414 30051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:30.089561 30051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:30.089731 30051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:30.091939 30051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6a797e8a534274b29ef79651d40e53" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41443 } }
I20260809 06:51:30.092473 30051 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6a797e8a534274b29ef79651d40e53; no voters:
I20260809 06:51:30.093056 30051 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:30.093187 30065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:30.093861 30065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6a797e8a534274b29ef79651d40e53, State: Running, Role: LEADER
I20260809 06:51:30.094362 30065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6a797e8a534274b29ef79651d40e53" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41443 } }
I20260809 06:51:30.095052 30051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:30.096871 30068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6a797e8a534274b29ef79651d40e53. Latest consensus state: current_term: 1 leader_uuid: "8f6a797e8a534274b29ef79651d40e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6a797e8a534274b29ef79651d40e53" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41443 } } }
I20260809 06:51:30.097198 30068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:30.097863 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6a797e8a534274b29ef79651d40e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6a797e8a534274b29ef79651d40e53" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41443 } } }
I20260809 06:51:30.098114 30071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:30.098214 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:30.102638 30071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:30.112048 30071 catalog_manager.cc:1347] Generated new cluster ID: 8ad79936dc4b4ae2be53474c38aad0e7
I20260809 06:51:30.112164 30071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:30.145814 30071 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:30.147365 30071 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:30.152268 30071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6a797e8a534274b29ef79651d40e53: Generated new TSK 0
I20260809 06:51:30.153009 30071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:30.164713 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29952
W20260809 06:51:30.165715 30041 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55543
W20260809 06:51:30.166316 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 29952: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:30.166442 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 29952
2026-08-09T06:51:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:30 dist-test-slave-jpj1 krb5kdc[30099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:30 dist-test-slave-jpj1 krb5kdc[30099](info): set up 2 sockets
Aug 09 06:51:30 dist-test-slave-jpj1 krb5kdc[30099](info): commencing operation
krb5kdc: starting...
W20260809 06:51:31.378854 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:31 dist-test-slave-jpj1 krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:31.676735 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41643
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:32.010622 30123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:32.010934 30123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:32.011029 30123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:32.018324 30123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:32.018455 30123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:32.018512 30123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:32.018573 30123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:32.018617 30123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:32.028605 30123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:41643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:32.029924 30123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:32.031265 30123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:32.042943 30132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:32.042896 30134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:32.042917 30137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:32.043762 30123 server_base.cc:1034] running on GCE node
I20260809 06:51:32.044590 30123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:32.046008 30123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:32.047217 30123 hybrid_clock.cc:648] HybridClock initialized: now 1786258292047169 us; error 43 us; skew 500 ppm
I20260809 06:51:32.047601 30123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:32 dist-test-slave-jpj1 krb5kdc[30099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:32.053722 30123 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:32.055313 30123 webserver.cc:460] Webserver started at http://127.27.209.254:44863/ using document root <none> and password file <none>
I20260809 06:51:32.056057 30123 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:32.056185 30123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:32.056494 30123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:32.059572 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "d8a4f48dc842484fb4e7db84314b7bf7"
format_stamp: "Formatted at 2026-08-09 06:51:32 on dist-test-slave-jpj1"
server_key: "d79cd34d76783c3478b7b32a0506d255"
server_key_iv: "e3383d75dadd0c2cc76c8b0261687882"
server_key_version: "encryptionkey@0"
I20260809 06:51:32.060374 30123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "d8a4f48dc842484fb4e7db84314b7bf7"
format_stamp: "Formatted at 2026-08-09 06:51:32 on dist-test-slave-jpj1"
server_key: "d79cd34d76783c3478b7b32a0506d255"
server_key_iv: "e3383d75dadd0c2cc76c8b0261687882"
server_key_version: "encryptionkey@0"
I20260809 06:51:32.065716 30123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:51:32.069298 30153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:32.070648 30123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 06:51:32.070814 30123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "d8a4f48dc842484fb4e7db84314b7bf7"
format_stamp: "Formatted at 2026-08-09 06:51:32 on dist-test-slave-jpj1"
server_key: "d79cd34d76783c3478b7b32a0506d255"
server_key_iv: "e3383d75dadd0c2cc76c8b0261687882"
server_key_version: "encryptionkey@0"
I20260809 06:51:32.071005 30123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:32.076915 30123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:32.080874 30123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:32.081153 30123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:32.099419 30123 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:41643
I20260809 06:51:32.099464 30225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:41643 every 8 connection(s)
I20260809 06:51:32.100889 30123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:32.104626 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30123
I20260809 06:51:32.104722 30226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:32.104805 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:32.105208 28487 external_mini_cluster.cc:1428] Setting key fdb6f9675c52161e529d99002f2cf87f
I20260809 06:51:32.113945 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:32.106981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33589 (local address 127.27.209.254:41643)
0809 06:51:32.107899 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:32.107919 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:32.107974 (+ 55us) server_negotiation.cc:407] Connection header received
0809 06:51:32.109912 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:32.109945 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:32.110418 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:32.110880 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:32.111945 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:32.111957 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:32.111974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:32.111993 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:32.112061 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:32.112100 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:32.112330 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:32.112475 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 06:51:32.112692 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":330,"threads_started":1}
I20260809 06:51:32.117542 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:32.119793 30226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7: Bootstrap starting.
I20260809 06:51:32.123651 30226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:32.124868 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:32.122574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54147 (local address 127.27.209.254:41643)
0809 06:51:32.122762 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:32.122768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:32.123129 (+ 361us) server_negotiation.cc:407] Connection header received
0809 06:51:32.123255 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:32.123260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:32.123321 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:32.123430 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:32.124287 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:32.124293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:32.124297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 06:51:32.124315 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:32.124362 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 06:51:32.124381 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:32.124493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:32.124536 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 06:51:32.124593 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 06:51:32.124943 30226 log.cc:824] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:32.127939 30226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7: No bootstrap required, opened a new log
I20260809 06:51:32.135250 30226 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41643 } }
I20260809 06:51:32.135669 30226 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 06:51:32.135773 30226 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a4f48dc842484fb4e7db84314b7bf7, State: Initialized, Role: FOLLOWER
I20260809 06:51:32.136360 30226 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41643 } }
I20260809 06:51:32.136579 30226 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 06:51:32.136713 30226 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 06:51:32.136860 30226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:32.138978 30226 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41643 } }
I20260809 06:51:32.139474 30226 leader_election.cc:302] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a4f48dc842484fb4e7db84314b7bf7; no voters:
I20260809 06:51:32.140190 30226 leader_election.cc:288] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 06:51:32.140326 30243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:32.140977 30243 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [term 1 LEADER]: Becoming Leader. State: Replica: d8a4f48dc842484fb4e7db84314b7bf7, State: Running, Role: LEADER
I20260809 06:51:32.141507 30243 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41643 } }
I20260809 06:51:32.142263 30226 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:32.144152 30245 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41643 } } }
I20260809 06:51:32.144595 30245 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:32.144181 30246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a4f48dc842484fb4e7db84314b7bf7. Latest consensus state: current_term: 1 leader_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a4f48dc842484fb4e7db84314b7bf7" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 41643 } } }
I20260809 06:51:32.145551 30246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7 [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:32.149456 30252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:32.154706 30252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:32.164072 30252 catalog_manager.cc:1347] Generated new cluster ID: 784b3dd52fe54029b8df9326d7eb1336
I20260809 06:51:32.164189 30252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:32.193482 30252 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:32.194013 30252 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:32.213948 30252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a4f48dc842484fb4e7db84314b7bf7: Generated new TSK 0
I20260809 06:51:32.215016 30252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:32.232043 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30123
W20260809 06:51:32.233275 30218 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33589
W20260809 06:51:32.234006 28487 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 06:51:32.234158 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30123
2026-08-09T06:51:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2068 ms)
[----------] 18 tests from AuthTokenIssuingTest (37411 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:32 dist-test-slave-jpj1 krb5kdc[30274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:32 dist-test-slave-jpj1 krb5kdc[30274](info): set up 2 sockets
Aug 09 06:51:32 dist-test-slave-jpj1 krb5kdc[30274](info): commencing operation
krb5kdc: starting...
W20260809 06:51:33.452328 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:33 dist-test-slave-jpj1 krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:33.752530 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:42175
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:42175,127.27.209.253:45931,127.27.209.252:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:34.089848 30294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:51:34.090219 30294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:34.090318 30294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:34.090435 30294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:34.097738 30294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:34.097846 30294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:34.097939 30294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:34.098006 30294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:34.098093 30294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:34.107981 30294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:42175,127.27.209.253:45931,127.27.209.252:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:42175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:34.109287 30294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:34.110647 30294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:34.122339 30304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:34.122373 30305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:34.122292 30309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:34.123570 30294 server_base.cc:1034] running on GCE node
I20260809 06:51:34.124369 30294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:34.125764 30294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:34.126994 30294 hybrid_clock.cc:648] HybridClock initialized: now 1786258294126922 us; error 42 us; skew 500 ppm
I20260809 06:51:34.127336 30294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:34 dist-test-slave-jpj1 krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:34.130529 30294 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:34.132030 30294 webserver.cc:460] Webserver started at http://127.27.209.254:44055/ using document root <none> and password file <none>
I20260809 06:51:34.132723 30294 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:34.132828 30294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:34.133167 30294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:34.136188 30294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a"
format_stamp: "Formatted at 2026-08-09 06:51:34 on dist-test-slave-jpj1"
server_key: "88e8da9a9bacaacaaab67afb9817fd68"
server_key_iv: "cd171086b16671ed89630475f2317ffa"
server_key_version: "encryptionkey@0"
I20260809 06:51:34.137029 30294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a"
format_stamp: "Formatted at 2026-08-09 06:51:34 on dist-test-slave-jpj1"
server_key: "88e8da9a9bacaacaaab67afb9817fd68"
server_key_iv: "cd171086b16671ed89630475f2317ffa"
server_key_version: "encryptionkey@0"
I20260809 06:51:34.142061 30294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 06:51:34.145560 30318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:34.146852 30294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 06:51:34.147023 30294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a"
format_stamp: "Formatted at 2026-08-09 06:51:34 on dist-test-slave-jpj1"
server_key: "88e8da9a9bacaacaaab67afb9817fd68"
server_key_iv: "cd171086b16671ed89630475f2317ffa"
server_key_version: "encryptionkey@0"
I20260809 06:51:34.147248 30294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:34.165172 30294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:34.169474 30294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:34.169741 30294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:34.187260 30294 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:42175
I20260809 06:51:34.187290 30401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:42175 every 8 connection(s)
I20260809 06:51:34.188570 30294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:34.190083 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30294
I20260809 06:51:34.190276 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:34.190979 28487 external_mini_cluster.cc:1428] Setting key a2c2f0b0b18680e0809c50d1b23dd742
I20260809 06:51:34.192322 30402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:34.198845 30402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:34 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.209.253@KRBTEST.COM; defaulting to no policy
I20260809 06:51:34.221047 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.202056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.27.209.254:42175)
0809 06:51:34.202835 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:34.202853 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:34.203269 (+ 416us) server_negotiation.cc:407] Connection header received
0809 06:51:34.205218 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:34.205230 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:34.205447 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:34.205621 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:34.206868 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.208086 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.209399 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.209941 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.210108 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.213461 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:34.213472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:34.213487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:34.213529 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:34.217020 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.217695 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.217704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.217708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.217918 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.218279 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.218284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.218287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.218971 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:34.219186 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:34.219764 (+ 578us) server_negotiation.cc:299] Negotiation successful
0809 06:51:34.220108 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":537,"thread_start_us":338,"threads_started":1}
I20260809 06:51:34.221081 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.202058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:42175 (local address 127.0.0.1:37496)
0809 06:51:34.203116 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.203155 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:34.204881 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:34.205658 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:34.205665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:34.206341 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:34.206667 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.206684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.208230 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.208234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.209272 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.209284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.210162 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.210169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.210395 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.211012 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:34.211033 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:34.213205 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:34.217157 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.217168 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.217176 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.217553 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.218024 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.218027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.218029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.218174 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.219120 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:34.219126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:34.219338 (+ 212us) client_negotiation.cc:769] Sending connection context
0809 06:51:34.219565 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 06:51:34.219947 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":45,"thread_start_us":657,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.253@KRBTEST.COM" created.
I20260809 06:51:34.224047 30402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:34.224982 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.224615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56290)
0809 06:51:34.224772 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.224878 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 06:51:34.225248 30322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
I20260809 06:51:34.226766 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.226409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56302)
0809 06:51:34.226573 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.226649 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 06:51:34.226967 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:45931: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab.
Entry for principal kudu/127.27.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab.
I20260809 06:51:34.258116 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:45931
--webserver_interface=127.27.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab
--principal=kudu/127.27.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:42175,127.27.209.253:45931,127.27.209.252:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:51:34.276275 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } attempt: 1
I20260809 06:51:34.277483 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.277035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56318)
0809 06:51:34.277227 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.277348 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:51:34.278548 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.278163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56322)
0809 06:51:34.278332 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.278403 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260809 06:51:34.278745 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:45931: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
I20260809 06:51:34.347000 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } attempt: 2
I20260809 06:51:34.348367 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.347811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56328)
0809 06:51:34.348078 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.348189 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:51:34.349356 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.348995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56330)
0809 06:51:34.349138 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.349228 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 06:51:34.349543 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:45931: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
I20260809 06:51:34.440825 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } attempt: 3
I20260809 06:51:34.442026 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.441566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56340)
0809 06:51:34.441785 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.441884 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 06:51:34.443023 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.442656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56356)
0809 06:51:34.442807 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.442878 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:51:34.443226 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:45931: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
I20260809 06:51:34.586447 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } attempt: 4
I20260809 06:51:34.587620 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.587074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56360)
0809 06:51:34.587356 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.587445 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260809 06:51:34.588573 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.588233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56368)
0809 06:51:34.588376 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.588457 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 06:51:34.588763 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:45931: Network error: Client connection negotiation failed: client connection to 127.27.209.253:45931: connect: Connection refused (error 111)
W20260809 06:51:34.589840 30413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:51:34.590157 30413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:34.590250 30413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:34.590364 30413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:34.597503 30413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:34.597591 30413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:34.597692 30413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:34.597757 30413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:34.597841 30413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:34.607527 30413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:42175,127.27.209.253:45931,127.27.209.252:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab
--principal=kudu/127.27.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:45931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:34.608816 30413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:34.610090 30413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:34.621248 30419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:34.621201 30420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:34.621423 30425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:34.622207 30413 server_base.cc:1034] running on GCE node
I20260809 06:51:34.622959 30413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:34.624329 30413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:34.625541 30413 hybrid_clock.cc:648] HybridClock initialized: now 1786258294625476 us; error 34 us; skew 500 ppm
I20260809 06:51:34.625854 30413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:34 dist-test-slave-jpj1 krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:34.628829 30413 init.cc:377] Logged in from keytab as kudu/127.27.209.253@KRBTEST.COM (short username kudu)
I20260809 06:51:34.630157 30413 webserver.cc:460] Webserver started at http://127.27.209.253:38947/ using document root <none> and password file <none>
I20260809 06:51:34.630810 30413 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:34.630901 30413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:34.631209 30413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:34.634083 30413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data/instance:
uuid: "369db92518d4437ab8f026eb5d3f8792"
format_stamp: "Formatted at 2026-08-09 06:51:34 on dist-test-slave-jpj1"
server_key: "9c6e4e0c78793161525aa556a0eef141"
server_key_iv: "d4261584d5e27e3408191d06118e66c6"
server_key_version: "encryptionkey@0"
I20260809 06:51:34.634878 30413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal/instance:
uuid: "369db92518d4437ab8f026eb5d3f8792"
format_stamp: "Formatted at 2026-08-09 06:51:34 on dist-test-slave-jpj1"
server_key: "9c6e4e0c78793161525aa556a0eef141"
server_key_iv: "d4261584d5e27e3408191d06118e66c6"
server_key_version: "encryptionkey@0"
I20260809 06:51:34.639726 30413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260809 06:51:34.642966 30438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:34.644356 30413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:34.644522 30413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal
uuid: "369db92518d4437ab8f026eb5d3f8792"
format_stamp: "Formatted at 2026-08-09 06:51:34 on dist-test-slave-jpj1"
server_key: "9c6e4e0c78793161525aa556a0eef141"
server_key_iv: "d4261584d5e27e3408191d06118e66c6"
server_key_version: "encryptionkey@0"
I20260809 06:51:34.644742 30413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:34.658286 30413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:34.662256 30413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:34.662495 30413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:34.678844 30413 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.253:45931
I20260809 06:51:34.678856 30521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.253:45931 every 8 connection(s)
I20260809 06:51:34.680109 30413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
I20260809 06:51:34.683781 30522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:34.688195 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30413
I20260809 06:51:34.688386 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-1/wal/instance
I20260809 06:51:34.688925 28487 external_mini_cluster.cc:1428] Setting key b644642652531b4b78708f7c8ac4db6b
I20260809 06:51:34.691082 30522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:34 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:34.711491 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.694155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.27.209.254:42175)
0809 06:51:34.694361 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:34.694367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:34.695601 (+ 1234us) server_negotiation.cc:407] Connection header received
0809 06:51:34.697498 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:34.697503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:34.697583 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:34.697695 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:34.699182 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.700114 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.701585 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.702155 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.702286 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.706132 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:34.706137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:34.706140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:34.706175 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:34.708657 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.709348 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.709352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.709355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.709473 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.709862 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.709867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.709869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.710142 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:34.710278 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:34.710950 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 06:51:34.711174 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:51:34.712440 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.694623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:42175 (local address 127.0.0.1:37500)
0809 06:51:34.695422 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.695466 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:34.697233 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:34.697924 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:34.697932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:34.698468 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:34.699007 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.699025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.700271 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.700275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.701434 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.701447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.702389 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.702396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.702687 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.703403 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:34.703433 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:34.705915 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:34.708788 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.708797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.708813 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.709236 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.709584 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.709589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.709593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.709758 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.710272 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:34.710280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:34.710583 (+ 303us) client_negotiation.cc:769] Sending connection context
0809 06:51:34.710829 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 06:51:34.711320 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":322,"threads_started":1}
I20260809 06:51:34.714347 30522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.252@KRBTEST.COM" created.
Aug 09 06:51:34 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:51:34.728753 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.714983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56378)
0809 06:51:34.715256 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.715275 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:34.715378 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:34.716346 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:34.716351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:34.716820 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:34.717015 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.717027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.718463 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.718469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.719683 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.719696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.720448 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.720452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.720586 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.721088 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:34.721105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:34.723402 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:34.726325 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.726330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.726333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.726773 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.727114 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.727118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.727121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.727278 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.727876 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:34.727880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:34.728026 (+ 146us) client_negotiation.cc:769] Sending connection context
0809 06:51:34.728156 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 06:51:34.728408 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260809 06:51:34.728854 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.715265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.27.209.253:45931)
0809 06:51:34.715937 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:34.715943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:34.715970 (+ 27us) server_negotiation.cc:407] Connection header received
0809 06:51:34.716011 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:34.716017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:34.716175 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:34.716293 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:34.717197 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.718318 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.719818 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.720331 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.720424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.723630 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:34.723642 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:34.723655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:34.723683 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:34.726189 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.726893 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.726899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.726902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.727020 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.727381 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.727385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.727387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.727755 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:34.727903 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:34.728277 (+ 374us) server_negotiation.cc:299] Negotiation successful
0809 06:51:34.728513 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":531,"thread_start_us":369,"threads_started":1}
I20260809 06:51:34.731217 30522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:34.732967 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.731987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57358)
0809 06:51:34.732610 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.732794 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260809 06:51:34.733258 30442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:34.735173 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.734765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57364)
0809 06:51:34.734947 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.735040 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 06:51:34.735651 30522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab.
Entry for principal kudu/127.27.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab.
I20260809 06:51:34.758490 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:46719
--webserver_interface=127.27.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab
--principal=kudu/127.27.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:42175,127.27.209.253:45931,127.27.209.252:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:51:34.785019 30522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 1
I20260809 06:51:34.786340 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.785781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57366)
0809 06:51:34.786012 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.786143 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 06:51:34.787366 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.786988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57376)
0809 06:51:34.787140 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.787234 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 06:51:34.787596 30522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:34.855809 30522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 2
I20260809 06:51:34.857019 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.856469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57392)
0809 06:51:34.856711 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.856829 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 06:51:34.857987 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.857623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57402)
0809 06:51:34.857764 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.857848 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 06:51:34.858193 30522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:34.887990 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } attempt: 5
Aug 09 06:51:34 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:51:34.900753 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.888801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.27.209.253:45931)
0809 06:51:34.888972 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:34.888978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:34.889049 (+ 71us) server_negotiation.cc:407] Connection header received
0809 06:51:34.889252 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:34.889256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:34.889325 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:34.889411 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:34.890217 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.890959 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.892211 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.892654 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.892777 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.895786 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:34.895791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:34.895794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:34.895828 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:34.898250 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.898855 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.898859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.898862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.898985 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:34.899344 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:34.899349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:34.899353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:34.899686 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:34.899831 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:34.900213 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 06:51:34.900428 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:51:34.900743 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.888736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56382)
0809 06:51:34.888946 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.888967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:34.889112 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:34.889459 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:34.889464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:34.889880 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:34.890062 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.890072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.891092 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.891097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.892062 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:34.892075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:34.892795 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:34.892801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:34.893003 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:34.893570 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:34.893588 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:34.895610 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:34.898382 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.898388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.898392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.898746 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.899088 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:34.899093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:34.899097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:34.899244 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:34.899805 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:34.899809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:34.899964 (+ 155us) client_negotiation.cc:769] Sending connection context
0809 06:51:34.900087 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 06:51:34.900377 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:51:34.902140 30402 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:34.903182 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.902810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57406)
0809 06:51:34.902968 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.903043 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 06:51:34.904111 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.903796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57412)
0809 06:51:34.903932 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.904018 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:51:34.904318 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:34.949415 30522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 3
I20260809 06:51:34.950608 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.950061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57428)
0809 06:51:34.950291 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.950412 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:51:34.951792 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.951282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57444)
0809 06:51:34.951519 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.951638 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260809 06:51:34.952035 30522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:34.955511 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 1
I20260809 06:51:34.956447 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.956055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57458)
0809 06:51:34.956221 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.956303 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 06:51:34.957551 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:34.957183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57474)
0809 06:51:34.957354 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:34.957429 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260809 06:51:34.957813 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:35.026032 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 2
I20260809 06:51:35.027261 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.026771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57488)
0809 06:51:35.026998 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.027088 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 06:51:35.028334 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.027964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57490)
0809 06:51:35.028147 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.028230 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 06:51:35.028549 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
W20260809 06:51:35.092298 30536 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:51:35.092626 30536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:35.092698 30536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:35.092777 30536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 06:51:35.095264 30522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 4
I20260809 06:51:35.096477 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.095934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57502)
0809 06:51:35.096183 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.096289 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 06:51:35.097497 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.097121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57510)
0809 06:51:35.097271 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.097359 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 06:51:35.097707 30522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
W20260809 06:51:35.100526 30536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:35.100606 30536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:35.100674 30536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:35.100723 30536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:35.100787 30536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:35.110364 30536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:42175,127.27.209.253:45931,127.27.209.252:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab
--principal=kudu/127.27.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:46719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:35.111620 30536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:35.112918 30536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:35.124330 30541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:35.124383 30544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:35.124490 30548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:35.125170 30536 server_base.cc:1034] running on GCE node
I20260809 06:51:35.125984 30536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:35.127358 30536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:35.128572 30536 hybrid_clock.cc:648] HybridClock initialized: now 1786258295128490 us; error 52 us; skew 500 ppm
I20260809 06:51:35.128909 30536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:35.131996 30536 init.cc:377] Logged in from keytab as kudu/127.27.209.252@KRBTEST.COM (short username kudu)
I20260809 06:51:35.133374 30536 webserver.cc:460] Webserver started at http://127.27.209.252:44787/ using document root <none> and password file <none>
I20260809 06:51:35.134104 30536 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:35.134191 30536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:35.134495 30536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:35.134747 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 3
I20260809 06:51:35.135906 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.135342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57514)
0809 06:51:35.135618 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.135723 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:51:35.136924 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.136523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57522)
0809 06:51:35.136702 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.136788 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260809 06:51:35.137157 30402 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:46719: Network error: Client connection negotiation failed: client connection to 127.27.209.252:46719: connect: Connection refused (error 111)
I20260809 06:51:35.138073 30536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data/instance:
uuid: "a9ffa99c29834a2fa590dbf2be210419"
format_stamp: "Formatted at 2026-08-09 06:51:35 on dist-test-slave-jpj1"
server_key: "6f3860a5ae35eda7d578f99005e3ff30"
server_key_iv: "fef89256a65b019ba2a0e60189cc4c30"
server_key_version: "encryptionkey@0"
I20260809 06:51:35.138809 30536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal/instance:
uuid: "a9ffa99c29834a2fa590dbf2be210419"
format_stamp: "Formatted at 2026-08-09 06:51:35 on dist-test-slave-jpj1"
server_key: "6f3860a5ae35eda7d578f99005e3ff30"
server_key_iv: "fef89256a65b019ba2a0e60189cc4c30"
server_key_version: "encryptionkey@0"
I20260809 06:51:35.143843 30536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 06:51:35.147104 30563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:35.148504 30536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 06:51:35.148643 30536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal
uuid: "a9ffa99c29834a2fa590dbf2be210419"
format_stamp: "Formatted at 2026-08-09 06:51:35 on dist-test-slave-jpj1"
server_key: "6f3860a5ae35eda7d578f99005e3ff30"
server_key_iv: "fef89256a65b019ba2a0e60189cc4c30"
server_key_version: "encryptionkey@0"
I20260809 06:51:35.148841 30536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:35.161156 30536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:35.165309 30536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:35.165560 30536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:35.183084 30536 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.252:46719
I20260809 06:51:35.183091 30649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.252:46719 every 8 connection(s)
I20260809 06:51:35.184422 30536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
I20260809 06:51:35.186082 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30536
I20260809 06:51:35.186254 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786257935933888-28487-0/minicluster-data/master-2/wal/instance
I20260809 06:51:35.186815 28487 external_mini_cluster.cc:1428] Setting key 45124a8f841fc78dff52d3ba2fc9d51a
I20260809 06:51:35.188807 30650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:35.198958 30650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:51:35.216825 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.201824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.27.209.254:42175)
0809 06:51:35.201991 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.201997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.202567 (+ 570us) server_negotiation.cc:407] Connection header received
0809 06:51:35.202912 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.202917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.202992 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.203077 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.204349 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.205046 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.206470 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.206905 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.207017 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.210600 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.210607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.210611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.210693 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.213601 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.214421 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.214427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.214429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.214569 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.214986 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.214990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.214994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.215279 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.215426 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.216206 (+ 780us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.216441 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:51:35.217984 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.188713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.27.209.252:46719)
0809 06:51:35.199930 (+ 11217us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.199946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.200191 (+ 245us) server_negotiation.cc:407] Connection header received
0809 06:51:35.202309 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.202334 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.202743 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.202950 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.204386 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.205524 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.206740 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.207265 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.207377 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.211717 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.211737 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.211749 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.211778 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.214560 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.215209 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.215215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.215217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.215335 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.215743 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.215747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.215749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.216098 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.216852 (+ 754us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.217336 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.217520 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":10971,"thread_start_us":10756,"threads_started":1}
I20260809 06:51:35.218065 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.201831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:42175 (local address 127.0.0.1:37504)
0809 06:51:35.202436 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.202469 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:35.202695 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:35.203166 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:35.203173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:35.203810 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:35.204181 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.204198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.205200 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.205210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.206330 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.206346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.207134 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.207141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.207355 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.208083 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:35.208104 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:35.210397 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:35.213743 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.213755 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.213768 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.214274 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.214700 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.214704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.214706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.214875 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.215421 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:35.215429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:35.215814 (+ 385us) client_negotiation.cc:769] Sending connection context
0809 06:51:35.216086 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 06:51:35.216609 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":37,"spinlock_wait_cycles":14336,"thread_start_us":287,"threads_started":1}
I20260809 06:51:35.220429 30650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:35.234143 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.221183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:45931 (local address 127.0.0.1:56384)
0809 06:51:35.221407 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.221425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:35.221536 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:35.221917 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:35.221922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:35.222240 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:35.222370 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.222378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.223650 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.223656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.224759 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.224772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.225704 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.225709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.225850 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.226419 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:35.226443 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:35.228688 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:35.231854 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.231860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.231864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.232335 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.232680 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.232685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.232689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.232866 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.233342 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:35.233347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:35.233505 (+ 158us) client_negotiation.cc:769] Sending connection context
0809 06:51:35.233681 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 06:51:35.233911 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260809 06:51:35.234197 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.221270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.27.209.253:45931)
0809 06:51:35.221440 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.221445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.221469 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:51:35.221708 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.221711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.221776 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.221875 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.222515 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.223461 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.224903 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.225555 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.225688 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.228879 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.228886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.228891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.228930 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.231733 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.232439 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.232443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.232445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.232573 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.232960 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.232963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.232965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.233229 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.233335 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.233747 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.233912 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 06:51:35.235152 30650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:35.236953 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.221448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37510 (local address 127.27.209.254:42175)
0809 06:51:35.221602 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.221607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.221624 (+ 17us) server_negotiation.cc:407] Connection header received
0809 06:51:35.221671 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.221676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.221743 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.221811 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.223234 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.224240 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.225850 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.226370 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.226497 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.230702 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.230706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.230710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.230742 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.233582 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.234936 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.234941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.234945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.235101 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.235458 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.235463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.235466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.235856 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.236313 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.236430 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.236652 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:51:35.249209 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.236029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57540)
0809 06:51:35.236272 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.236286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:35.236386 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:35.236636 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:35.236640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:35.237091 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:35.237272 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.237283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.238873 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.238879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.239969 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.239986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.240927 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.240933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.241154 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.241684 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:35.241714 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:35.243890 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:35.246731 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.246736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.246739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.247250 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.247616 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.247621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.247625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.247784 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.248486 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:35.248489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:35.248602 (+ 113us) client_negotiation.cc:769] Sending connection context
0809 06:51:35.248717 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 06:51:35.248917 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260809 06:51:35.249500 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.236072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.27.209.252:46719)
0809 06:51:35.236231 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.236236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.236350 (+ 114us) server_negotiation.cc:407] Connection header received
0809 06:51:35.236474 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.236479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.236536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.236608 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.237824 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.238742 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.240123 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.240617 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.240722 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.244063 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.244068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.244071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.244098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.246594 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.247355 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.247359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.247361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.247482 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.247905 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.247910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.247914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.248279 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.248449 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.248846 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.249077 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 06:51:35.255007 30650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: Bootstrap starting.
I20260809 06:51:35.255143 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.240249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.27.209.253:45931)
0809 06:51:35.240395 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.240399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.240413 (+ 14us) server_negotiation.cc:407] Connection header received
0809 06:51:35.240454 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.240457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.240509 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.240581 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.242091 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.242797 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.244328 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.244749 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.244857 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.249623 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.249629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.249633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.249699 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.252603 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.253262 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.253267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.253270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.253467 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.253813 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.253818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.253821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.254169 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.254487 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.254616 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.254838 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 06:51:35.258733 30650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:35.259801 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:35.259954 30650 log.cc:824] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:35.262717 30650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: No bootstrap required, opened a new log
I20260809 06:51:35.262537 30365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37510
I20260809 06:51:35.273413 30650 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.273865 30650 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ffa99c29834a2fa590dbf2be210419, State: Initialized, Role: FOLLOWER
I20260809 06:51:35.274394 30650 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.275396 30660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.275774 30660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.276571 30650 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 06:51:35.286079 30683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:51:35.286219 30683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:51:35.284080 30536 master_runner.cc:186] Error getting master registration for 127.27.209.254:42175: OK, instance_id { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" instance_seqno: 1786258294182310 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:51:35.294952 30536 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:51:35.314401 30402 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 4
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:51:35.331497 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.315248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57552)
0809 06:51:35.315486 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.315509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:35.315686 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:35.316055 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:35.316060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:35.316619 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:35.316849 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.316861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.318344 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.318350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.319875 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.319893 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.320940 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.320947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.321182 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.321864 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:35.321889 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:35.324721 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:35.328583 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.328589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.328593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.329133 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.329544 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.329549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.329553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.329750 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.330466 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:35.330479 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:35.330660 (+ 181us) client_negotiation.cc:769] Sending connection context
0809 06:51:35.330805 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 06:51:35.331079 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 06:51:35.331600 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.315310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.27.209.252:46719)
0809 06:51:35.315523 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.315527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.315601 (+ 74us) server_negotiation.cc:407] Connection header received
0809 06:51:35.315812 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.315818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.315899 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.316005 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.317010 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.318174 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.320031 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.320774 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.320898 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.324904 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.324910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.324914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.324946 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.328448 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.329273 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.329279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.329283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.329437 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.329875 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.329881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.329884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.330358 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.330533 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.330955 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.331209 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 06:51:35.338191 30402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a: Bootstrap starting.
I20260809 06:51:35.342512 30402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:35.343895 30402 log.cc:824] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:35.347342 30402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a: No bootstrap required, opened a new log
I20260809 06:51:35.355222 30402 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.355666 30402 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0f8f8063bf4b578f333a9ebb7bd52a, State: Initialized, Role: FOLLOWER
I20260809 06:51:35.356197 30402 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.357225 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.357599 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.358485 30402 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:35.358681 30365 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 06:51:35.358863 30365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:35.361341 30365 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.362592 30365 leader_election.cc:288] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [CANDIDATE]: Term 1 election: Requested vote from peers 369db92518d4437ab8f026eb5d3f8792 (127.27.209.253:45931), a9ffa99c29834a2fa590dbf2be210419 (127.27.209.252:46719)
I20260809 06:51:35.364319 30603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9ffa99c29834a2fa590dbf2be210419"
I20260809 06:51:35.364802 30603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:35.368232 30603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd0f8f8063bf4b578f333a9ebb7bd52a in term 1.
I20260809 06:51:35.368973 30319 leader_election.cc:302] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ffa99c29834a2fa590dbf2be210419, fd0f8f8063bf4b578f333a9ebb7bd52a; no voters:
I20260809 06:51:35.369455 30686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:35.382377 30686 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [term 1 LEADER]: Becoming Leader. State: Replica: fd0f8f8063bf4b578f333a9ebb7bd52a, State: Running, Role: LEADER
I20260809 06:51:35.383055 30686 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.391301 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0f8f8063bf4b578f333a9ebb7bd52a. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.391822 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:35.395790 30699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:35.396896 30522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } attempt: 5
Aug 09 06:51:35 dist-test-slave-jpj1 krb5kdc[30274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:51:35.412114 30699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:35.415416 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.397562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:46719 (local address 127.0.0.1:57568)
0809 06:51:35.398031 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:35.398053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:35.398200 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:35.399191 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:35.399197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:35.399746 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:35.399968 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.399982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.401542 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.401548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.402962 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:35.402980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.404207 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.404213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:35.404441 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.405131 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:35.405158 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:35.408103 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:35.412413 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.412418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.412423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.412935 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.413398 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:35.413403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:35.413407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:35.413596 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:35.414297 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:35.414303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:35.414457 (+ 154us) client_negotiation.cc:769] Sending connection context
0809 06:51:35.414695 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 06:51:35.414989 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260809 06:51:35.416440 30655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.398466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.27.209.252:46719)
0809 06:51:35.398664 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.398671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.398694 (+ 23us) server_negotiation.cc:407] Connection header received
0809 06:51:35.398760 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.398766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.398863 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.398979 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.400190 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.401354 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.403161 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.403870 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.403986 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.408348 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.408355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.408360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.408399 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.412255 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.413078 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.413084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.413088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.413265 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.413729 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.413734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.413741 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.414072 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.414230 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.415819 (+ 1589us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.416053 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 06:51:35.417559 30570 tablet.cc:2369] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:51:35.422260 30522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: Bootstrap starting.
W20260809 06:51:35.422363 30329 tablet.cc:2369] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:51:35.426311 30522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:35.427852 30522 log.cc:824] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:35.436511 30522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: No bootstrap required, opened a new log
I20260809 06:51:35.456875 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.417605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.27.209.252:46719)
0809 06:51:35.425794 (+ 8189us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.425801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.428265 (+ 2464us) server_negotiation.cc:407] Connection header received
0809 06:51:35.428327 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.428332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.428408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.428506 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.430783 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.432059 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.433997 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.434594 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.434721 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.439727 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.439734 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.439739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.439784 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.443342 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.444227 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.444233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.444236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.444400 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.452824 (+ 8424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.452830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.452834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.453508 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.455566 (+ 2058us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.455792 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.456191 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7355,"thread_start_us":6356,"threads_started":1}
I20260809 06:51:35.457424 30522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.457863 30522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 369db92518d4437ab8f026eb5d3f8792, State: Initialized, Role: FOLLOWER
I20260809 06:51:35.458427 30522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } }
I20260809 06:51:35.461146 30603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [term 1 FOLLOWER]: Refusing update from remote peer fd0f8f8063bf4b578f333a9ebb7bd52a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:51:35.462316 30692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:51:35.466202 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.422118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.27.209.253:45931)
0809 06:51:35.422300 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.422306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.431192 (+ 8886us) server_negotiation.cc:407] Connection header received
0809 06:51:35.432015 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.432022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.432103 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.432174 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.443995 (+ 11821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.445136 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.449548 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.450164 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.450291 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.457635 (+ 7344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.457641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.457646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.457686 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.461309 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.462649 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.462655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.462658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.462834 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.463496 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.463502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.463505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.463902 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.464142 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.465325 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.465563 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 06:51:35.470050 30716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.470715 30716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.471140 30522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:35.474107 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:35.410079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.27.209.254:42175)
0809 06:51:35.410263 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.410269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.423504 (+ 13235us) server_negotiation.cc:407] Connection header received
0809 06:51:35.424310 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.424317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.424412 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.424488 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.436210 (+ 11722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.437324 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.450648 (+ 13324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:35.451600 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:35.451720 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:35.455294 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:35.455301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:35.455305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:35.455357 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:35.459204 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.470715 (+ 11511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.470721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.470724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.470963 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:35.471883 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:35.471889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:35.471892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:35.472277 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:35.472546 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:35.473450 (+ 904us) server_negotiation.cc:299] Negotiation successful
0809 06:51:35.473707 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 06:51:35.471194 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "369db92518d4437ab8f026eb5d3f8792"
I20260809 06:51:35.477221 30480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:35.481199 30480 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [term 1 FOLLOWER]: Refusing update from remote peer fd0f8f8063bf4b578f333a9ebb7bd52a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:51:35.484767 30481 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd0f8f8063bf4b578f333a9ebb7bd52a in term 1.
I20260809 06:51:35.485646 30660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0f8f8063bf4b578f333a9ebb7bd52a. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.485981 30660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.486984 30692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [LEADER]: Connected to new peer: Peer: permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:51:35.498533 30720 mvcc.cc:205] Tried to move back new op lower bound from 7316513978161971200 to 7316513977896058880. Current Snapshot: MvccSnapshot[applied={T|T < 7316513978161971200}]
I20260809 06:51:35.517369 30660 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.517658 30660 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.518991 30699 catalog_manager.cc:1347] Generated new cluster ID: 442b86b97120446aa0e261f29d32263b
I20260809 06:51:35.519110 30699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:35.526654 30716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0f8f8063bf4b578f333a9ebb7bd52a. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.527019 30716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.537031 30716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.537308 30716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:35.538000 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.538308 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:35.548012 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0f8f8063bf4b578f333a9ebb7bd52a" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 42175 } } peers { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 45931 } } peers { permanent_uuid: "a9ffa99c29834a2fa590dbf2be210419" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 46719 } } }
I20260809 06:51:35.537130 30294 master_runner.cc:186] Error getting master registration for 127.27.209.253:45931: OK, instance_id { permanent_uuid: "369db92518d4437ab8f026eb5d3f8792" instance_seqno: 1786258294674137 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:51:35.548372 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:35.548552 30294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:51:35.550140 30751 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 06:51:35.550539 30699 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:35.552086 30699 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:35.564700 30699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd0f8f8063bf4b578f333a9ebb7bd52a: Generated new TSK 0
I20260809 06:51:35.565954 30699 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 06:51:35.642520 30405 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.629172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.27.209.254:42175)
0809 06:51:35.629378 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.629385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.640309 (+ 10924us) server_negotiation.cc:407] Connection header received
0809 06:51:35.640628 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.640635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.640768 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.640861 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.642256 (+ 1395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37532: BlockingRecv error: recv got EOF from 127.0.0.1:37532 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 06:51:35.642830 30705 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.629308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.27.209.252:46719)
0809 06:51:35.629508 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.629514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.640062 (+ 10548us) server_negotiation.cc:407] Connection header received
0809 06:51:35.641357 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.641362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.641443 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.641517 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.642593 (+ 1076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57590: BlockingRecv error: recv got EOF from 127.0.0.1:57590 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 06:51:35.645234 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.640640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.27.209.253:45931)
0809 06:51:35.641133 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.641139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.643444 (+ 2305us) server_negotiation.cc:407] Connection header received
0809 06:51:35.643701 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.643707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.643798 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.643866 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.644990 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56416: BlockingRecv error: recv got EOF from 127.0.0.1:56416 (error 108)
Metrics: {"server-negotiator.queue_time_us":349}
W20260809 06:51:35.652765 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.650781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.27.209.253:45931)
0809 06:51:35.650938 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.650944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.651374 (+ 430us) server_negotiation.cc:407] Connection header received
0809 06:51:35.651592 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.651598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.651718 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.651818 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.652531 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56432: BlockingRecv error: recv got EOF from 127.0.0.1:56432 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 06:51:35.673038 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.656779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.27.209.253:45931)
0809 06:51:35.656928 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.656934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.671671 (+ 14737us) server_negotiation.cc:407] Connection header received
0809 06:51:35.671879 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.671884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.671971 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.672067 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.672827 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56442: BlockingRecv error: recv got EOF from 127.0.0.1:56442 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 06:51:35.685720 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.678774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.27.209.253:45931)
0809 06:51:35.678956 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.678963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.684083 (+ 5120us) server_negotiation.cc:407] Connection header received
0809 06:51:35.684287 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.684292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.684377 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.684469 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.685476 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56450: BlockingRecv error: recv got EOF from 127.0.0.1:56450 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:51:35.718796 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.716285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.27.209.253:45931)
0809 06:51:35.716528 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.716534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.717068 (+ 534us) server_negotiation.cc:407] Connection header received
0809 06:51:35.717269 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.717275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.717375 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.717511 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.718557 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56466: BlockingRecv error: recv got EOF from 127.0.0.1:56466 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260809 06:51:35.733402 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.731174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.27.209.253:45931)
0809 06:51:35.731354 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.731360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.731953 (+ 593us) server_negotiation.cc:407] Connection header received
0809 06:51:35.732146 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.732152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.732245 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.732351 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.733164 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56470: BlockingRecv error: recv got EOF from 127.0.0.1:56470 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 06:51:35.759835 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.757417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.27.209.253:45931)
0809 06:51:35.757609 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.757624 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.758184 (+ 560us) server_negotiation.cc:407] Connection header received
0809 06:51:35.758374 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.758380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.758513 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.758630 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.759563 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56482: BlockingRecv error: recv got EOF from 127.0.0.1:56482 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 06:51:35.798645 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.796234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.27.209.253:45931)
0809 06:51:35.796426 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.796433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.797043 (+ 610us) server_negotiation.cc:407] Connection header received
0809 06:51:35.797249 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.797255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.797349 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.797463 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.798346 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56488: BlockingRecv error: recv got EOF from 127.0.0.1:56488 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260809 06:51:35.868929 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:35.866857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.27.209.253:45931)
0809 06:51:35.867089 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:35.867096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:35.867496 (+ 400us) server_negotiation.cc:407] Connection header received
0809 06:51:35.867708 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:35.867714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:35.867819 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:35.867957 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:35.868697 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56498: BlockingRecv error: recv got EOF from 127.0.0.1:56498 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 06:51:36.010958 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:36.008906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.27.209.253:45931)
0809 06:51:36.009143 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.009150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.009496 (+ 346us) server_negotiation.cc:407] Connection header received
0809 06:51:36.009748 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.009753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.009846 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.009937 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:36.010692 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56512: BlockingRecv error: recv got EOF from 127.0.0.1:56512 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 06:51:36.272977 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:36.271168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.27.209.253:45931)
0809 06:51:36.271353 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.271358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.271637 (+ 279us) server_negotiation.cc:407] Connection header received
0809 06:51:36.271852 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.271856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.271936 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.272015 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:36.272800 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56516: BlockingRecv error: recv got EOF from 127.0.0.1:56516 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 06:51:36.288329 30683 catalog_manager.cc:1259] Loaded cluster ID: 442b86b97120446aa0e261f29d32263b
I20260809 06:51:36.288429 30683 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: loading cluster ID for follower catalog manager: success
I20260809 06:51:36.290913 30683 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: acquiring CA information for follower catalog manager: success
I20260809 06:51:36.292814 30683 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9ffa99c29834a2fa590dbf2be210419: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:51:36.554670 30751 catalog_manager.cc:1259] Loaded cluster ID: 442b86b97120446aa0e261f29d32263b
I20260809 06:51:36.554788 30751 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: loading cluster ID for follower catalog manager: success
I20260809 06:51:36.557474 30751 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: acquiring CA information for follower catalog manager: success
I20260809 06:51:36.559427 30751 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 369db92518d4437ab8f026eb5d3f8792: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:51:36.796383 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:36.789251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.27.209.253:45931)
0809 06:51:36.789996 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.790000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.790021 (+ 21us) server_negotiation.cc:407] Connection header received
0809 06:51:36.790084 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.790088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.790235 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.790339 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:36.790855 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.791909 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.793912 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.794370 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.794471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:36.795055 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:36.795724 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:36.795834 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:36.795983 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 06:51:36.796047 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":494,"threads_started":1}
I20260809 06:51:36.808882 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:36.803264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.27.209.252:46719)
0809 06:51:36.803937 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.803942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.803973 (+ 31us) server_negotiation.cc:407] Connection header received
0809 06:51:36.804072 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.804076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.804233 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.804390 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:36.804781 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.805493 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.806794 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.807177 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.807295 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:36.807772 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:36.808275 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:36.808406 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:36.808531 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 06:51:36.808609 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":429,"threads_started":1}
I20260809 06:51:36.832291 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:36.820158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.27.209.254:42175)
0809 06:51:36.820977 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.820983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.824095 (+ 3112us) server_negotiation.cc:407] Connection header received
0809 06:51:36.824474 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.824481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.824728 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.824909 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:36.825767 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.826990 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.829351 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.830037 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.830138 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:36.830942 (+ 804us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:36.831450 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:36.831566 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:36.831811 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 06:51:36.831914 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":582,"threads_started":1}
I20260809 06:51:36.841260 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:36.820438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.27.209.252:46719)
0809 06:51:36.820622 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.820627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.832366 (+ 11739us) server_negotiation.cc:407] Connection header received
0809 06:51:36.833014 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.833020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.833146 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.833219 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:36.833981 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.835045 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.838573 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.839158 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.839276 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:36.839768 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:36.839918 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:36.839984 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:36.840887 (+ 903us) server_negotiation.cc:299] Negotiation successful
0809 06:51:36.840978 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:51:36.841364 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:36.824227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.27.209.253:45931)
0809 06:51:36.824407 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:36.824412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:36.833070 (+ 8658us) server_negotiation.cc:407] Connection header received
0809 06:51:36.833366 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:36.833373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:36.833468 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:36.833529 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:36.834231 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.835386 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.839020 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:36.839733 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:36.839866 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:36.840477 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:36.840608 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:36.840711 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:36.840920 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 06:51:36.841007 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 06:51:36.841473 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30294
I20260809 06:51:36.842470 30395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:36.953871 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30294
I20260809 06:51:36.981079 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30413
I20260809 06:51:36.982266 30513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:37.086841 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30413
I20260809 06:51:37.111805 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30536
I20260809 06:51:37.112962 30635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:37.218703 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30536
2026-08-09T06:51:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4991 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 06:51:37 dist-test-slave-jpj1 krb5kdc[30893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 06:51:37 dist-test-slave-jpj1 krb5kdc[30893](info): set up 2 sockets
Aug 09 06:51:37 dist-test-slave-jpj1 krb5kdc[30893](info): commencing operation
krb5kdc: starting...
W20260809 06:51:38.469987 28487 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/test-user.keytab.
Aug 09 06:51:38 dist-test-slave-jpj1 krb5kdc[30893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T06:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T06:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.209.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
Entry for principal kudu/127.27.209.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab.
I20260809 06:51:38.766268 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44129
--webserver_interface=127.27.209.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44129,127.27.209.253:44171,127.27.209.252:40181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 06:51:39.102061 30917 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:51:39.102402 30917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:39.102483 30917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:39.102566 30917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:39.109686 30917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:39.109776 30917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:39.109839 30917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:39.109890 30917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:39.109952 30917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:39.119949 30917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44129,127.27.209.253:44171,127.27.209.252:40181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.254.keytab
--principal=kudu/127.27.209.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.254:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.209.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:39.121223 30917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:39.122617 30917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:39.134032 30923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:39.133986 30924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:39.133960 30926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:39.134980 30917 server_base.cc:1034] running on GCE node
I20260809 06:51:39.135784 30917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:39.137090 30917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:39.138319 30917 hybrid_clock.cc:648] HybridClock initialized: now 1786258299138247 us; error 35 us; skew 500 ppm
I20260809 06:51:39.138634 30917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:39 dist-test-slave-jpj1 krb5kdc[30893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:39.141712 30917 init.cc:377] Logged in from keytab as kudu/127.27.209.254@KRBTEST.COM (short username kudu)
I20260809 06:51:39.143129 30917 webserver.cc:460] Webserver started at http://127.27.209.254:44839/ using document root <none> and password file <none>
I20260809 06:51:39.143868 30917 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:39.143975 30917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:39.144284 30917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:39.147147 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data/instance:
uuid: "5b7b39a44066485f9f7a8b2a9f6577ed"
format_stamp: "Formatted at 2026-08-09 06:51:39 on dist-test-slave-jpj1"
server_key: "9fa5ff6d70324e14274b5ed6eb316ba8"
server_key_iv: "2c922e5db04cdaa536c71f94dccea702"
server_key_version: "encryptionkey@0"
I20260809 06:51:39.147958 30917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal/instance:
uuid: "5b7b39a44066485f9f7a8b2a9f6577ed"
format_stamp: "Formatted at 2026-08-09 06:51:39 on dist-test-slave-jpj1"
server_key: "9fa5ff6d70324e14274b5ed6eb316ba8"
server_key_iv: "2c922e5db04cdaa536c71f94dccea702"
server_key_version: "encryptionkey@0"
I20260809 06:51:39.152920 30917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:51:39.156312 30935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:39.157608 30917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 06:51:39.157778 30917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal
uuid: "5b7b39a44066485f9f7a8b2a9f6577ed"
format_stamp: "Formatted at 2026-08-09 06:51:39 on dist-test-slave-jpj1"
server_key: "9fa5ff6d70324e14274b5ed6eb316ba8"
server_key_iv: "2c922e5db04cdaa536c71f94dccea702"
server_key_version: "encryptionkey@0"
I20260809 06:51:39.157976 30917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:39.183310 30917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:39.187254 30917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:39.187508 30917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:39.205152 30917 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.254:44129
I20260809 06:51:39.205128 31017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.254:44129 every 8 connection(s)
I20260809 06:51:39.206525 30917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/data/info.pb
I20260809 06:51:39.210170 31018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:39.215428 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 30917
I20260809 06:51:39.215620 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-0/wal/instance
I20260809 06:51:39.216024 28487 external_mini_cluster.cc:1428] Setting key b58fd5475a18643e0d6174fcc11b4182
I20260809 06:51:39.216581 31018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:39 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:39.237931 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.219754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44129 (local address 127.0.0.1:59712)
0809 06:51:39.220494 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.220533 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:39.222531 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:39.223313 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:39.223319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:39.224020 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:39.224352 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.224370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.225845 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.225848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.226880 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.226892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.227798 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.227805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.227986 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.228606 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:39.228628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:39.230799 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:39.233861 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.233872 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.233884 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.234284 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.234678 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.234683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.234687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.234844 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.235591 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:39.235603 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:39.235806 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 06:51:39.236034 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 06:51:39.236657 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":159,"thread_start_us":296,"threads_started":1}
I20260809 06:51:39.237967 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.220950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.27.209.254:44129)
0809 06:51:39.221630 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:39.221638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:39.221698 (+ 60us) server_negotiation.cc:407] Connection header received
0809 06:51:39.222871 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:39.222876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:39.223100 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:39.223278 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:39.224549 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.225696 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.227009 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.227577 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.227749 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.231010 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:39.231017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:39.231032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:39.231061 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:39.233738 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.234405 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.234411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.234414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.234564 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.234946 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.234950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.234952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.235429 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:39.235648 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:39.236167 (+ 519us) server_negotiation.cc:299] Negotiation successful
0809 06:51:39.236531 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260809 06:51:39.241151 31018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:39.242270 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.241899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56230)
0809 06:51:39.242054 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.242159 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 06:51:39.242559 30939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
I20260809 06:51:39.244092 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.243756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56246)
0809 06:51:39.243895 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.243960 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.27.209.253@KRBTEST.COM; defaulting to no policy
W20260809 06:51:39.244302 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:44171: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab.
Entry for principal kudu/127.27.209.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab.
I20260809 06:51:39.280968 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:44171
--webserver_interface=127.27.209.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab
--principal=kudu/127.27.209.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44129,127.27.209.253:44171,127.27.209.252:40181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:51:39.293610 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } attempt: 1
I20260809 06:51:39.294701 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.294251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56260)
0809 06:51:39.294451 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.294568 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 06:51:39.295737 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.295358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56262)
0809 06:51:39.295499 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.295596 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 06:51:39.295910 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:44171: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
I20260809 06:51:39.364142 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } attempt: 2
I20260809 06:51:39.365257 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.364781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56276)
0809 06:51:39.365023 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.365112 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 06:51:39.366319 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.365960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56280)
0809 06:51:39.366115 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.366183 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 06:51:39.366487 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:44171: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
I20260809 06:51:39.457739 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } attempt: 3
I20260809 06:51:39.458918 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.458426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56292)
0809 06:51:39.458677 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.458779 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 06:51:39.460031 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.459674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56302)
0809 06:51:39.459819 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.459892 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 06:51:39.460220 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:44171: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
I20260809 06:51:39.603482 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } attempt: 4
I20260809 06:51:39.604657 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.604121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56310)
0809 06:51:39.604428 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.604514 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260809 06:51:39.605640 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.605307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56320)
0809 06:51:39.605446 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.605520 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 06:51:39.605806 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.253:44171: Network error: Client connection negotiation failed: client connection to 127.27.209.253:44171: connect: Connection refused (error 111)
W20260809 06:51:39.619133 31029 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:51:39.619467 31029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:39.619576 31029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:39.619712 31029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 06:51:39.626860 31029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:39.626947 31029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:39.627033 31029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:39.627096 31029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:39.627180 31029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:39.636780 31029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44129,127.27.209.253:44171,127.27.209.252:40181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.253.keytab
--principal=kudu/127.27.209.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.253:44171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.209.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:39.637996 31029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:39.639318 31029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 06:51:39.650405 31038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:39.650425 31040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:39.650486 31044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:39.651391 31029 server_base.cc:1034] running on GCE node
I20260809 06:51:39.652112 31029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:39.653345 31029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:39.654557 31029 hybrid_clock.cc:648] HybridClock initialized: now 1786258299654496 us; error 32 us; skew 500 ppm
I20260809 06:51:39.654870 31029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:39 dist-test-slave-jpj1 krb5kdc[30893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:39.657833 31029 init.cc:377] Logged in from keytab as kudu/127.27.209.253@KRBTEST.COM (short username kudu)
I20260809 06:51:39.659113 31029 webserver.cc:460] Webserver started at http://127.27.209.253:42283/ using document root <none> and password file <none>
I20260809 06:51:39.659839 31029 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:39.659943 31029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:39.660226 31029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:39.663201 31029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data/instance:
uuid: "5431439afef44be0ba6c6a09a49dcde9"
format_stamp: "Formatted at 2026-08-09 06:51:39 on dist-test-slave-jpj1"
server_key: "563fa5fb282fed3a3df75a188ebd9f71"
server_key_iv: "f312ec5cefa98948a75d3b78c18b1328"
server_key_version: "encryptionkey@0"
I20260809 06:51:39.663980 31029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal/instance:
uuid: "5431439afef44be0ba6c6a09a49dcde9"
format_stamp: "Formatted at 2026-08-09 06:51:39 on dist-test-slave-jpj1"
server_key: "563fa5fb282fed3a3df75a188ebd9f71"
server_key_iv: "f312ec5cefa98948a75d3b78c18b1328"
server_key_version: "encryptionkey@0"
I20260809 06:51:39.668757 31029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 06:51:39.672003 31053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:39.673219 31029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 06:51:39.673390 31029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal
uuid: "5431439afef44be0ba6c6a09a49dcde9"
format_stamp: "Formatted at 2026-08-09 06:51:39 on dist-test-slave-jpj1"
server_key: "563fa5fb282fed3a3df75a188ebd9f71"
server_key_iv: "f312ec5cefa98948a75d3b78c18b1328"
server_key_version: "encryptionkey@0"
I20260809 06:51:39.673579 31029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:39.690707 31029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:39.694439 31029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:39.694679 31029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:39.711426 31029 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.253:44171
I20260809 06:51:39.711450 31123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.253:44171 every 8 connection(s)
I20260809 06:51:39.712682 31029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/data/info.pb
I20260809 06:51:39.716346 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:39.720458 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 31029
I20260809 06:51:39.720646 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-1/wal/instance
I20260809 06:51:39.721181 28487 external_mini_cluster.cc:1428] Setting key 7c158fd10205c71017dd7032a497b55b
I20260809 06:51:39.723958 31126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:39 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:39.746309 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.726979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.27.209.254:44129)
0809 06:51:39.727194 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:39.727201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:39.728408 (+ 1207us) server_negotiation.cc:407] Connection header received
0809 06:51:39.730713 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:39.730718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:39.730800 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:39.730877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:39.732814 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.733473 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.735461 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.736118 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.736243 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.739747 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:39.739752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:39.739756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:39.739788 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:39.742587 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.743636 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.743642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.743645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.743814 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.744305 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.744310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.744313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.744721 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:39.744879 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:39.745719 (+ 840us) server_negotiation.cc:299] Negotiation successful
0809 06:51:39.745965 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 06:51:39.747373 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.727459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44129 (local address 127.0.0.1:59720)
0809 06:51:39.728231 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.728278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:39.730409 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:39.731135 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:39.731147 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:39.731898 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:39.732626 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.732667 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.733630 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.733635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.735294 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.735310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.736315 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.736320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.736517 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.737161 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:39.737186 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:39.739478 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:39.742791 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.742817 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.742835 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.743457 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.743953 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.743956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.743959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.744165 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.744859 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:39.744867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:39.745273 (+ 406us) client_negotiation.cc:769] Sending connection context
0809 06:51:39.745558 (+ 285us) client_negotiation.cc:240] Negotiation successful
0809 06:51:39.746077 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":282,"threads_started":1}
I20260809 06:51:39.752167 31126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.209.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.209.252@KRBTEST.COM" created.
Aug 09 06:51:39 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:51:39.767139 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.753046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56336)
0809 06:51:39.753241 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.753262 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:39.753389 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:39.754719 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:39.754724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:39.755170 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:39.755314 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.755323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.756775 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.756779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.757629 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.757644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.758482 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.758488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.758659 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.759183 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:39.759200 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:39.761485 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:39.764666 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.764671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.764674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.765112 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.765436 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.765440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.765443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.765636 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.766250 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:39.766254 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:39.766397 (+ 143us) client_negotiation.cc:769] Sending connection context
0809 06:51:39.766531 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 06:51:39.766798 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 06:51:39.767266 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.753592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.27.209.253:44171)
0809 06:51:39.754304 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:39.754310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:39.754338 (+ 28us) server_negotiation.cc:407] Connection header received
0809 06:51:39.754385 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:39.754390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:39.754572 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:39.754711 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:39.755477 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.756626 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.757774 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.758346 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.758462 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.761684 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:39.761699 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:39.761717 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:39.761751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:39.764539 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.765226 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.765233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.765235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.765347 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.765732 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.765736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.765738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.766126 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:39.766303 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:39.766665 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 06:51:39.766924 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":542,"thread_start_us":318,"threads_started":1}
I20260809 06:51:39.769958 31126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:39.771174 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.770715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59250)
0809 06:51:39.770880 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.771014 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 06:51:39.771438 31056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:39.772971 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.772651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59252)
0809 06:51:39.772810 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.772879 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:51:39.773169 31126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.209.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab.
Entry for principal kudu/127.27.209.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab.
I20260809 06:51:39.795032 28487 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
/tmp/dist-test-taskHh9PjI/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:40181
--webserver_interface=127.27.209.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab
--principal=kudu/127.27.209.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.209.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.209.254:44129,127.27.209.253:44171,127.27.209.252:40181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 06:51:39.822448 31126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 1
I20260809 06:51:39.823608 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.823130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59258)
0809 06:51:39.823318 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.823427 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 06:51:39.824548 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.824199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59274)
0809 06:51:39.824366 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.824444 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 06:51:39.824739 31126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:39.892951 31126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 2
I20260809 06:51:39.894232 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.893758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59284)
0809 06:51:39.893968 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.894070 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 06:51:39.895200 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.894859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59300)
0809 06:51:39.895002 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.895075 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 06:51:39.895408 31126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:39.905043 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } attempt: 5
Aug 09 06:51:39 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:51:39.918108 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.905826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56338)
0809 06:51:39.906045 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.906064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:39.906196 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:39.906554 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:39.906559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:39.906972 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:39.907194 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.907217 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.908243 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.908249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.909110 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:39.909123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.909875 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.909881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:39.910071 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.910642 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:39.910660 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:39.912906 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:39.915790 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.915795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.915800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.916146 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.916485 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:39.916490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:39.916494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:39.916638 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:39.917128 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:39.917133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:39.917295 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 06:51:39.917434 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 06:51:39.917753 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 06:51:39.918112 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.905837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.27.209.253:44171)
0809 06:51:39.906011 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:39.906017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:39.906130 (+ 113us) server_negotiation.cc:407] Connection header received
0809 06:51:39.906331 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:39.906335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:39.906415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:39.906501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:39.907354 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.908099 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.909267 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:39.909726 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:39.909843 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:39.913104 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:39.913109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:39.913112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:39.913144 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:39.915642 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.916253 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.916258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.916262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.916381 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:39.916739 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:39.916744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:39.916748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:39.917016 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:39.917156 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:39.917568 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 06:51:39.917773 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 06:51:39.919524 31018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } has no permanent_uuid. Determining permanent_uuid...
I20260809 06:51:39.920795 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.920388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59316)
0809 06:51:39.920554 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.920661 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 06:51:39.921821 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.921477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59318)
0809 06:51:39.921651 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.921727 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 06:51:39.922029 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:39.973237 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 1
I20260809 06:51:39.974472 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.973970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59324)
0809 06:51:39.974203 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.974299 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260809 06:51:39.975456 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.975150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59326)
0809 06:51:39.975272 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.975345 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 06:51:39.975690 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:39.986617 31126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 3
I20260809 06:51:39.987679 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.987172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59342)
0809 06:51:39.987348 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.987459 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 06:51:39.988875 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:39.988414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59354)
0809 06:51:39.988657 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:39.988743 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260809 06:51:39.989104 31126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:40.043917 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 2
I20260809 06:51:40.045212 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.044667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59364)
0809 06:51:40.044925 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.045021 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 06:51:40.046232 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.045884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59376)
0809 06:51:40.046032 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.046114 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 06:51:40.046440 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
W20260809 06:51:40.125869 31136 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 06:51:40.126225 31136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 06:51:40.126322 31136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 06:51:40.126431 31136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 06:51:40.132298 31126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 4
I20260809 06:51:40.133411 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.132953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59378)
0809 06:51:40.133157 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.133252 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 06:51:40.133965 31136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 06:51:40.134050 31136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 06:51:40.134142 31136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 06:51:40.134223 31136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 06:51:40.134315 31136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 06:51:40.134465 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.134061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59390)
0809 06:51:40.134234 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.134333 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 06:51:40.134724 31126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
I20260809 06:51:40.144233 31136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.209.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.209.254:44129,127.27.209.253:44171,127.27.209.252:40181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/krb5kdc/kudu_127.27.209.252.keytab
--principal=kudu/127.27.209.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.209.252:40181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.209.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 05:54:04 UTC on ccycloud.quasar-xkgyaq.root.comops.site
build id 4575853
ASAN enabled
I20260809 06:51:40.145476 31136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 06:51:40.146786 31136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 06:51:40.152642 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 3
I20260809 06:51:40.153654 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.153233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59398)
0809 06:51:40.153416 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.153498 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 06:51:40.154645 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.154283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59412)
0809 06:51:40.154468 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.154536 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260809 06:51:40.154874 31018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.209.252:40181: Network error: Client connection negotiation failed: client connection to 127.27.209.252:40181: connect: Connection refused (error 111)
W20260809 06:51:40.158311 31146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:40.158396 31147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 06:51:40.158360 31150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 06:51:40.159123 31136 server_base.cc:1034] running on GCE node
I20260809 06:51:40.160005 31136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 06:51:40.161393 31136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 06:51:40.162616 31136 hybrid_clock.cc:648] HybridClock initialized: now 1786258300162536 us; error 46 us; skew 500 ppm
I20260809 06:51:40.162926 31136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 06:51:40.165879 31136 init.cc:377] Logged in from keytab as kudu/127.27.209.252@KRBTEST.COM (short username kudu)
I20260809 06:51:40.167243 31136 webserver.cc:460] Webserver started at http://127.27.209.252:37217/ using document root <none> and password file <none>
I20260809 06:51:40.167936 31136 fs_manager.cc:359] Metadata directory not provided
I20260809 06:51:40.168025 31136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 06:51:40.168300 31136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 06:51:40.171121 31136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data/instance:
uuid: "a6de44df45cb4eafb4c7532a00e8d2f9"
format_stamp: "Formatted at 2026-08-09 06:51:40 on dist-test-slave-jpj1"
server_key: "4ba8085619896e27087fcfa93eef19c5"
server_key_iv: "b4e16adef2ccf07143ba7e6f3cd61570"
server_key_version: "encryptionkey@0"
I20260809 06:51:40.171911 31136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal/instance:
uuid: "a6de44df45cb4eafb4c7532a00e8d2f9"
format_stamp: "Formatted at 2026-08-09 06:51:40 on dist-test-slave-jpj1"
server_key: "4ba8085619896e27087fcfa93eef19c5"
server_key_iv: "b4e16adef2ccf07143ba7e6f3cd61570"
server_key_version: "encryptionkey@0"
I20260809 06:51:40.176807 31136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 06:51:40.180240 31160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 06:51:40.181510 31136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 06:51:40.181648 31136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data,/tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal
uuid: "a6de44df45cb4eafb4c7532a00e8d2f9"
format_stamp: "Formatted at 2026-08-09 06:51:40 on dist-test-slave-jpj1"
server_key: "4ba8085619896e27087fcfa93eef19c5"
server_key_iv: "b4e16adef2ccf07143ba7e6f3cd61570"
server_key_version: "encryptionkey@0"
I20260809 06:51:40.181825 31136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 06:51:40.197165 31136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 06:51:40.200843 31136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 06:51:40.201058 31136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 06:51:40.220518 31136 rpc_server.cc:307] RPC server started. Bound to: 127.27.209.252:40181
I20260809 06:51:40.220538 31236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.209.252:40181 every 8 connection(s)
I20260809 06:51:40.221833 31136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/data/info.pb
I20260809 06:51:40.222227 28487 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu as pid 31136
I20260809 06:51:40.222401 28487 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskHh9PjI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786257935933888-28487-0/minicluster-data/master-2/wal/instance
I20260809 06:51:40.222951 28487 external_mini_cluster.cc:1428] Setting key 6182227c33a3440d2255e58314c533ef
I20260809 06:51:40.226153 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 06:51:40.234239 31237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
I20260809 06:51:40.245901 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.224790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.27.209.252:40181)
0809 06:51:40.225551 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.225562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.225604 (+ 42us) server_negotiation.cc:407] Connection header received
0809 06:51:40.227229 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.227252 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.227670 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.227985 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.229707 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.231000 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.232714 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.233369 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.233495 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.238269 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.238278 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.238292 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.238335 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.241646 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.242329 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.242336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.242341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.242496 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.242888 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.242893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.242896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.243589 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.243901 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.244265 (+ 364us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.244685 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":348,"threads_started":1}
I20260809 06:51:40.252305 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.237268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.254:44129 (local address 127.0.0.1:59734)
0809 06:51:40.238025 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.238074 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:40.238246 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:40.238621 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:40.238626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:40.239265 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:40.239581 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.239595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.240950 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.240954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.242015 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.242027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.242866 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.242872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.243045 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.243621 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:40.243648 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:40.246164 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:40.249805 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.249816 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.249823 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.250384 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.250733 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.250738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.250742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.250912 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.251447 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:40.251453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:40.251652 (+ 199us) client_negotiation.cc:769] Sending connection context
0809 06:51:40.251811 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 06:51:40.252023 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":364,"threads_started":1}
I20260809 06:51:40.252553 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.237290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.27.209.254:44129)
0809 06:51:40.237492 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.237499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.238172 (+ 673us) server_negotiation.cc:407] Connection header received
0809 06:51:40.238388 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.238393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.238474 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.238565 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.239751 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.240790 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.242160 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.242708 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.242834 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.246403 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.246408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.246411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.246446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.249663 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.250500 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.250504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.250506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.250623 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.251024 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.251028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.251030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.251325 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.251482 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.251966 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.252180 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 06:51:40.253898 31237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.254@KRBTEST.COM
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
I20260809 06:51:40.265672 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.248966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.27.209.254:44129)
0809 06:51:40.249611 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.249617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.249643 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:51:40.249685 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.249689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.249756 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.249847 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.251325 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.252442 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.254060 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.254607 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.254731 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.259510 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.259517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.259522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.259604 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.262782 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.263674 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.263678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.263680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.263811 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.264211 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.264215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.264217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.264513 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.264780 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.265077 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.265303 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":259,"threads_started":1}
I20260809 06:51:40.268837 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.254625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.253:44171 (local address 127.0.0.1:56350)
0809 06:51:40.254817 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.254862 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:40.254982 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:40.255662 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:40.255666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:40.256087 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:40.256271 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.256282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.257600 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.257606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.258623 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.258636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.259565 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.259572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.259785 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.260280 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:40.260302 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:40.262782 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:40.266085 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.266090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.266094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.266447 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.266876 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.266881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.266884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.267058 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.267956 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:40.267960 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:40.268136 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 06:51:40.268294 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 06:51:40.268597 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 06:51:40.269062 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.254912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.27.209.253:44171)
0809 06:51:40.255094 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.255100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.255126 (+ 26us) server_negotiation.cc:407] Connection header received
0809 06:51:40.255179 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.255184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.255256 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.255345 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.256450 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.257456 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.258772 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.259402 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.259517 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.263025 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.263032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.263037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.263080 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.265954 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.266568 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.266574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.266577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.266760 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.267204 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.267209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.267213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.267712 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.267882 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.268469 (+ 587us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.268704 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 06:51:40.270072 31237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.209.253@KRBTEST.COM
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.252@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:51:40.283676 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.270922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59430)
0809 06:51:40.271085 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.271103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:40.271211 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:40.271465 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:40.271474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:40.271967 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:40.272128 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.272139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.273390 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.273414 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.274655 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.274669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.275702 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.275706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.275891 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.276394 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:40.276433 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:40.278404 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:40.281141 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.281146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.281151 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.281616 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.281976 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.281979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.281981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.282112 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.282649 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:40.282652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:40.282781 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 06:51:40.282919 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 06:51:40.283229 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 06:51:40.283953 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.271087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.27.209.252:40181)
0809 06:51:40.271238 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.271242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.271257 (+ 15us) server_negotiation.cc:407] Connection header received
0809 06:51:40.271296 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.271299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.271348 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.271423 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.272273 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.273229 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.274851 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.275472 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.275652 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.278563 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.278569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.278573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.278601 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.281005 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.281729 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.281733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.281736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.281858 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.282224 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.282227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.282229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.282533 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.282675 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.283027 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.283418 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":65}
I20260809 06:51:40.284274 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.267445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.27.209.253:44171)
0809 06:51:40.268095 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.268101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.268123 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:51:40.268169 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.268174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.268242 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.268336 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.270050 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.271085 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.272711 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.273433 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.273551 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.278471 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.278476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.278479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.278512 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.281202 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.281950 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.281955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.281959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.282091 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.282470 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.282474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.282477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.282800 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.283015 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.283360 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.283606 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":287,"threads_started":1}
I20260809 06:51:40.289053 28487 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 06:51:40.289428 31237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: Bootstrap starting.
I20260809 06:51:40.292606 31237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:40.293637 31237 log.cc:824] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:40.291236 30978 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59742
I20260809 06:51:40.296378 31237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: No bootstrap required, opened a new log
I20260809 06:51:40.303300 31237 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.303696 31237 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6de44df45cb4eafb4c7532a00e8d2f9, State: Initialized, Role: FOLLOWER
I20260809 06:51:40.304128 31237 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.305040 31246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.305410 31246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.305923 31237 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 06:51:40.315032 31264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 06:51:40.315166 31264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 06:51:40.313081 31136 master_runner.cc:186] Error getting master registration for 127.27.209.254:44129: OK, instance_id { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" instance_seqno: 1786258299199984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:51:40.323892 31136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:51:40.332043 31018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 4
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.254@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:51:40.346892 31020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.332845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59440)
0809 06:51:40.333056 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.333078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:40.333220 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:40.333575 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:40.333581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:40.333989 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:40.334161 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.334171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.335167 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.335173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.336159 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.336172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.336957 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.336963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.337156 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.337772 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:40.337791 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:40.340020 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:40.344052 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.344058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.344063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.344582 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.344989 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.344994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.344998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.345178 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.345875 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:40.345880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:40.346059 (+ 179us) client_negotiation.cc:769] Sending connection context
0809 06:51:40.346207 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 06:51:40.346492 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 06:51:40.346995 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.332901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.27.209.252:40181)
0809 06:51:40.333100 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.333105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.333129 (+ 24us) server_negotiation.cc:407] Connection header received
0809 06:51:40.333377 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.333382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.333445 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.333526 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.334320 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.335037 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.336310 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.336817 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.336927 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.340243 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.340250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.340255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.340298 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.343908 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.344707 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.344712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.344716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.344872 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.345284 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.345289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.345292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.345769 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.345944 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.346331 (+ 387us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.346553 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:51:40.353242 31018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed: Bootstrap starting.
I20260809 06:51:40.356470 31018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:40.357589 31018 log.cc:824] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:40.362051 31018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed: No bootstrap required, opened a new log
I20260809 06:51:40.367254 31018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.367672 31018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7b39a44066485f9f7a8b2a9f6577ed, State: Initialized, Role: FOLLOWER
I20260809 06:51:40.368122 31018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.368995 31268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.369411 31268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.370158 31018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:40.370312 30978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 06:51:40.370541 30978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:40.374068 30978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.376678 30978 leader_election.cc:288] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [CANDIDATE]: Term 1 election: Requested vote from peers 5431439afef44be0ba6c6a09a49dcde9 (127.27.209.253:44171), a6de44df45cb4eafb4c7532a00e8d2f9 (127.27.209.252:40181)
I20260809 06:51:40.376590 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9"
I20260809 06:51:40.377121 31200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:40.380677 31200 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7b39a44066485f9f7a8b2a9f6577ed in term 1.
I20260809 06:51:40.381332 30938 leader_election.cc:302] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7b39a44066485f9f7a8b2a9f6577ed, a6de44df45cb4eafb4c7532a00e8d2f9; no voters:
I20260809 06:51:40.381809 31268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 1 FOLLOWER]: Leader election won for term 1
I20260809 06:51:40.382473 31268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [term 1 LEADER]: Becoming Leader. State: Replica: 5b7b39a44066485f9f7a8b2a9f6577ed, State: Running, Role: LEADER
I20260809 06:51:40.383080 31268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.386942 31283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7b39a44066485f9f7a8b2a9f6577ed. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.387393 31283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:40.390398 31292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 06:51:40.405709 31292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 06:51:40.423990 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.390310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.27.209.253:44171)
0809 06:51:40.390512 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.390518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.398314 (+ 7796us) server_negotiation.cc:407] Connection header received
0809 06:51:40.398964 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.398971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.399070 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.399147 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.401811 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.402985 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.410794 (+ 7809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.411456 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.411610 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.416894 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.416901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.416906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.416953 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.420539 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.421536 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.421542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.421546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.421717 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.422197 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.422202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.422205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.422579 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.422821 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.423369 (+ 548us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.423633 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:51:40.433934 31126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } attempt: 5
I20260809 06:51:40.439574 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.389637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.27.209.254:44129)
0809 06:51:40.389833 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.389840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.409894 (+ 20054us) server_negotiation.cc:407] Connection header received
0809 06:51:40.410109 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.410116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.410213 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.410308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.413270 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.414355 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.422680 (+ 8325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.423322 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.423414 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.432128 (+ 8714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.432135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.432140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.432183 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.436038 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.436866 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.436871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.436875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.437282 (+ 407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.437774 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.437779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.437783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.438150 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.438391 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.438965 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.439199 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 06:51:40.443811 30942 tablet.cc:2369] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:51:40.444515 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.393781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.27.209.252:40181)
0809 06:51:40.393962 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.393968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.427062 (+ 33094us) server_negotiation.cc:407] Connection header received
0809 06:51:40.427449 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.427455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.427564 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.427633 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.429327 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.430442 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.432323 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.432977 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.433081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.436215 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.436222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.436227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.436269 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.440292 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.441118 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.441124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.441127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.441310 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.442519 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.442524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.442527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.442850 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.443123 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.443516 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.443917 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 06:51:40 dist-test-slave-jpj1 krb5kdc[30893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786258299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.209.253@KRBTEST.COM for kudu/127.27.209.252@KRBTEST.COM
I20260809 06:51:40.449143 31200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [term 1 FOLLOWER]: Refusing update from remote peer 5b7b39a44066485f9f7a8b2a9f6577ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:51:40.450695 31283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 06:51:40.452912 31170 tablet.cc:2369] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 06:51:40.458846 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.434449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.209.252:40181 (local address 127.0.0.1:59458)
0809 06:51:40.434804 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0809 06:51:40.434822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 06:51:40.434922 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 06:51:40.441342 (+ 6420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 06:51:40.441348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 06:51:40.441852 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0809 06:51:40.442058 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.442071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.443645 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.443651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.444955 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 06:51:40.444970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.446221 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.446227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 06:51:40.446463 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.447092 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 06:51:40.447114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 06:51:40.450898 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 06:51:40.456035 (+ 5137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.456040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.456043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.456523 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.456983 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 06:51:40.456987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 06:51:40.456990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 06:51:40.457190 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 06:51:40.457861 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 06:51:40.457866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 06:51:40.458032 (+ 166us) client_negotiation.cc:769] Sending connection context
0809 06:51:40.458187 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 06:51:40.458475 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260809 06:51:40.460821 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.435125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.27.209.252:40181)
0809 06:51:40.440983 (+ 5858us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.440988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.441010 (+ 22us) server_negotiation.cc:407] Connection header received
0809 06:51:40.441066 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.441070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.441145 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.441262 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.442221 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.443469 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.445139 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.446050 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.446789 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.451155 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 06:51:40.451161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 06:51:40.451166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 06:51:40.451201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 06:51:40.455884 (+ 4683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.456688 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.456693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.456697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.456860 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 06:51:40.457334 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 06:51:40.457338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 06:51:40.457342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 06:51:40.457717 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 06:51:40.460043 (+ 2326us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.460172 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.460410 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5716,"thread_start_us":5517,"threads_started":1}
I20260809 06:51:40.466467 31126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: Bootstrap starting.
I20260809 06:51:40.430497 30917 master_runner.cc:186] Error getting master registration for 127.27.209.253:44171: OK, instance_id { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" instance_seqno: 1786258299706682 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 06:51:40.469647 30917 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 06:51:40.470551 31126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: Neither blocks nor log segments found. Creating new log.
I20260809 06:51:40.472122 31126 log.cc:824] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: Log is configured to *not* fsync() on all Append() calls
I20260809 06:51:40.476711 31246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7b39a44066485f9f7a8b2a9f6577ed. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.477033 31246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.483006 31309 mvcc.cc:205] Tried to move back new op lower bound from 7316513998558732288 to 7316513998374694912. Current Snapshot: MvccSnapshot[applied={T|T < 7316513998558732288}]
I20260809 06:51:40.487452 31126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: No bootstrap required, opened a new log
I20260809 06:51:40.503675 31246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.503983 31246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.506656 31126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.507393 31126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5431439afef44be0ba6c6a09a49dcde9, State: Initialized, Role: FOLLOWER
I20260809 06:51:40.508380 31126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } }
I20260809 06:51:40.511168 31292 catalog_manager.cc:1347] Generated new cluster ID: bb30dc6161e44223b6d9f8713a791108
I20260809 06:51:40.511298 31292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 06:51:40.512629 31268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.512969 31268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:40.513928 31126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 06:51:40.514338 31087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 06:51:40.517889 31087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [term 1 FOLLOWER]: Refusing update from remote peer 5b7b39a44066485f9f7a8b2a9f6577ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 06:51:40.518520 31088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5431439afef44be0ba6c6a09a49dcde9"
I20260809 06:51:40.519150 31317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.519618 31317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.521926 31088 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7b39a44066485f9f7a8b2a9f6577ed in term 1.
I20260809 06:51:40.522846 31283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 06:51:40.564771 31292 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 06:51:40.566826 31292 catalog_manager.cc:1504] Loading token signing keys...
I20260809 06:51:40.591401 31317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7b39a44066485f9f7a8b2a9f6577ed. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.591764 31317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.592008 31268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.592304 31268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed [sys.catalog]: This master's current role is: LEADER
I20260809 06:51:40.597730 31292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b7b39a44066485f9f7a8b2a9f6577ed: Generated new TSK 0
I20260809 06:51:40.598690 31292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 06:51:40.595635 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7b39a44066485f9f7a8b2a9f6577ed" member_type: VOTER last_known_addr { host: "127.27.209.254" port: 44129 } } peers { permanent_uuid: "5431439afef44be0ba6c6a09a49dcde9" member_type: VOTER last_known_addr { host: "127.27.209.253" port: 44171 } } peers { permanent_uuid: "a6de44df45cb4eafb4c7532a00e8d2f9" member_type: VOTER last_known_addr { host: "127.27.209.252" port: 40181 } } }
I20260809 06:51:40.600703 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 06:51:40.618314 31337 catalog_manager.cc:1259] Loaded cluster ID: bb30dc6161e44223b6d9f8713a791108
I20260809 06:51:40.618413 31337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: loading cluster ID for follower catalog manager: success
I20260809 06:51:40.621646 31337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: acquiring CA information for follower catalog manager: success
I20260809 06:51:40.623930 31337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5431439afef44be0ba6c6a09a49dcde9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 06:51:40.748661 31243 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:40.728809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.27.209.253:44171)
0809 06:51:40.729004 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.729011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.745508 (+ 16497us) server_negotiation.cc:407] Connection header received
0809 06:51:40.747001 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.747009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.747125 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.747199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.748396 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56378: BlockingRecv error: recv got EOF from 127.0.0.1:56378 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 06:51:40.750694 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:40.730243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.27.209.252:40181)
0809 06:51:40.730434 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.730440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.746732 (+ 16292us) server_negotiation.cc:407] Connection header received
0809 06:51:40.749178 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.749184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.749298 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.749369 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.750484 (+ 1115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59462: BlockingRecv error: recv got EOF from 127.0.0.1:59462 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 06:51:40.753007 31242 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:40.743509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.27.209.254:44129)
0809 06:51:40.743726 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.743733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.751235 (+ 7502us) server_negotiation.cc:407] Connection header received
0809 06:51:40.751461 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.751466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.751570 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.751636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.752772 (+ 1136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47808: BlockingRecv error: recv got EOF from 127.0.0.1:47808 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 06:51:40.798436 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.782366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.27.209.253:44171)
0809 06:51:40.782565 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.782570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.790283 (+ 7713us) server_negotiation.cc:407] Connection header received
0809 06:51:40.790822 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.790829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.791060 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.791205 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:40.792013 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.793142 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.795219 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.795944 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.796046 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.796868 (+ 822us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:40.797581 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:40.797703 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.797968 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.798061 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 06:51:40.845127 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:40.831916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.27.209.252:40181)
0809 06:51:40.832121 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.832127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.842691 (+ 10564us) server_negotiation.cc:407] Connection header received
0809 06:51:40.843303 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.843310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.843459 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.843599 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.844829 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33172: BlockingRecv error: recv got EOF from 127.0.0.1:33172 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 06:51:40.895979 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.880965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.27.209.253:44171)
0809 06:51:40.881169 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.881176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.888823 (+ 7647us) server_negotiation.cc:407] Connection header received
0809 06:51:40.889344 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.889351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.889456 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.889528 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:40.890290 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.891411 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.893478 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.894220 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.894354 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.895025 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:40.895204 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:40.895276 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.895551 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.895642 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 06:51:40.943526 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:40.930495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.27.209.252:40181)
0809 06:51:40.930714 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.930720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.941355 (+ 10635us) server_negotiation.cc:407] Connection header received
0809 06:51:40.941890 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.941897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.942008 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.942095 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.943244 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33176: BlockingRecv error: recv got EOF from 127.0.0.1:33176 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 06:51:40.977914 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:40.970771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.27.209.253:44171)
0809 06:51:40.970970 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.970976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.971503 (+ 527us) server_negotiation.cc:407] Connection header received
0809 06:51:40.971753 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.971760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.971864 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.971973 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:40.972454 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.973631 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.975717 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:40.976383 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:40.976526 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:40.977098 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:40.977266 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:40.977356 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:40.977519 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 06:51:40.977622 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 06:51:40.997735 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:40.984826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.27.209.252:40181)
0809 06:51:40.985021 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:40.985026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:40.996114 (+ 11088us) server_negotiation.cc:407] Connection header received
0809 06:51:40.996364 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:40.996370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:40.996471 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:40.996581 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:40.997501 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33180: BlockingRecv error: recv got EOF from 127.0.0.1:33180 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 06:51:41.012010 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.005241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.27.209.253:44171)
0809 06:51:41.005417 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.005423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.005501 (+ 78us) server_negotiation.cc:407] Connection header received
0809 06:51:41.005851 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.005857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.005985 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.006081 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.006644 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.007906 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.009910 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.010576 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.010702 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.011171 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.011335 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.011417 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.011647 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.011723 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260809 06:51:41.020774 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:41.018798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.27.209.252:40181)
0809 06:51:41.019048 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.019055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.019352 (+ 297us) server_negotiation.cc:407] Connection header received
0809 06:51:41.019502 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.019508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.019653 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.019775 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:41.020519 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33194: BlockingRecv error: recv got EOF from 127.0.0.1:33194 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260809 06:51:41.036897 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.031698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.27.209.253:44171)
0809 06:51:41.031888 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.031895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.032264 (+ 369us) server_negotiation.cc:407] Connection header received
0809 06:51:41.032462 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.032466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.032556 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.032639 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.033051 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.033862 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.035190 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.035663 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.035807 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.036155 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.036265 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.036346 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.036484 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.036574 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 06:51:41.044366 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:41.042562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.27.209.252:40181)
0809 06:51:41.042777 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.042783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.043022 (+ 239us) server_negotiation.cc:407] Connection header received
0809 06:51:41.043164 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.043170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.043267 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.043386 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:41.044119 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33204: BlockingRecv error: recv got EOF from 127.0.0.1:33204 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 06:51:41.069531 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.063721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.27.209.253:44171)
0809 06:51:41.063933 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.063941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.064212 (+ 271us) server_negotiation.cc:407] Connection header received
0809 06:51:41.064389 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.064395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.064490 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.064602 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.064960 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.066151 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.067602 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.068290 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.068435 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.068817 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.068932 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.069023 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.069176 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.069257 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260809 06:51:41.079103 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:41.076934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.27.209.252:40181)
0809 06:51:41.077130 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.077137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.077626 (+ 489us) server_negotiation.cc:407] Connection header received
0809 06:51:41.077870 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.077874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.077946 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.078024 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:41.078881 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33206: BlockingRecv error: recv got EOF from 127.0.0.1:33206 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 06:51:41.120326 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.114564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.27.209.253:44171)
0809 06:51:41.114791 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.114797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.115185 (+ 388us) server_negotiation.cc:407] Connection header received
0809 06:51:41.115395 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.115399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.115472 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.115570 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.116018 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.116764 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.118665 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.119132 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.119227 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.119687 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.119804 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.119866 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.119998 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.120062 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 06:51:41.128355 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:41.126657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.27.209.252:40181)
0809 06:51:41.126820 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.126824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.127180 (+ 356us) server_negotiation.cc:407] Connection header received
0809 06:51:41.127354 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.127359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.127428 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.127504 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:41.128171 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33210: BlockingRecv error: recv got EOF from 127.0.0.1:33210 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 06:51:41.200269 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.195662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.27.209.253:44171)
0809 06:51:41.195862 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.195867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.196242 (+ 375us) server_negotiation.cc:407] Connection header received
0809 06:51:41.196423 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.196427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.196496 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.196580 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.196926 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.197654 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.198904 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.199304 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.199399 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.199721 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.199829 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.199893 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.199990 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.200048 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 06:51:41.207657 31307 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 06:51:41.205764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.27.209.252:40181)
0809 06:51:41.205953 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.205958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.206331 (+ 373us) server_negotiation.cc:407] Connection header received
0809 06:51:41.206479 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.206484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.206572 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.206711 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 06:51:41.207412 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33222: BlockingRecv error: recv got EOF from 127.0.0.1:33222 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 06:51:41.317193 31264 catalog_manager.cc:1259] Loaded cluster ID: bb30dc6161e44223b6d9f8713a791108
I20260809 06:51:41.317314 31264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: loading cluster ID for follower catalog manager: success
I20260809 06:51:41.319808 31264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: acquiring CA information for follower catalog manager: success
I20260809 06:51:41.321597 31264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6de44df45cb4eafb4c7532a00e8d2f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 06:51:41.343684 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.339058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.27.209.253:44171)
0809 06:51:41.339308 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.339315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.339517 (+ 202us) server_negotiation.cc:407] Connection header received
0809 06:51:41.339718 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.339722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.339811 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.339889 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.340230 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.340919 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.342201 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.342635 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.342722 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.343070 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.343173 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.343232 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.343332 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.343394 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 06:51:41.354456 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.349205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.27.209.252:40181)
0809 06:51:41.349363 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.349367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.349755 (+ 388us) server_negotiation.cc:407] Connection header received
0809 06:51:41.349906 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.349909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.350062 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.350153 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.350508 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.351230 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.352560 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.352964 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.353053 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.353474 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.353981 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.354086 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.354163 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.354229 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 06:51:41.367026 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.359813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.27.209.252:40181)
0809 06:51:41.359985 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.359990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.361152 (+ 1162us) server_negotiation.cc:407] Connection header received
0809 06:51:41.361378 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.361382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.361449 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.361529 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.362197 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.363104 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.365340 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.365740 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.365830 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.366297 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.366435 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.366523 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.366664 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.366744 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 06:51:41.367218 31243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.359806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.27.209.253:44171)
0809 06:51:41.359970 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.359976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.360866 (+ 890us) server_negotiation.cc:407] Connection header received
0809 06:51:41.361164 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.361170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.361332 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.361436 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.362207 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.363180 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.365401 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.365864 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.366002 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.366443 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.366581 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.366654 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.366874 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.366950 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 06:51:41.368498 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 06:51:41.359521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.27.209.254:44129)
0809 06:51:41.360019 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0809 06:51:41.360025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 06:51:41.361562 (+ 1537us) server_negotiation.cc:407] Connection header received
0809 06:51:41.361891 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 06:51:41.361896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 06:51:41.362159 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 06:51:41.362350 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 06:51:41.362893 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.363761 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.365900 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 06:51:41.366468 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 06:51:41.366573 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 06:51:41.367465 (+ 892us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 06:51:41.367905 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 06:51:41.368004 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 06:51:41.368118 (+ 114us) server_negotiation.cc:299] Negotiation successful
0809 06:51:41.368199 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":45440,"thread_start_us":308,"threads_started":1}
I20260809 06:51:41.373147 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 30917
I20260809 06:51:41.374059 31010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:41.489077 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 30917
I20260809 06:51:41.516956 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31029
I20260809 06:51:41.518050 31118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:41.623682 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 31029
I20260809 06:51:41.645963 28487 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu pid 31136
I20260809 06:51:41.647084 31230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 06:51:41.752135 28487 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskHh9PjI/build/asan/bin/kudu with pid 31136
2026-08-09T06:51:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4530 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9521 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (365787 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST